scikit learn transform

[Stephane-Thales]

Hi,
I'm trying to apply sklearn.preprocessing.OneHotEncoder to a datatable column.

preprocess_encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')
preprocess_encoder.fit(mydatatable[:,f['mycol']])
encoded = preprocess_encoder.transform(mydatatable[:,f['mycol']])

the fit work but the transform return TypeError: '_io.BufferedWriter' object is not subscriptable

Is there a way to make datatable "subscriptable" ?

[oleksiyskononenko]

Perhaps, sklearn just doesn’t support datatable frames as an input?

[oleksiyskononenko]

As a fast workaround, try to convert dt frame to numpy before passing it to sklearn: https://datatable.readthedocs.io/en/latest/api/frame/to_numpy.html

Also, consider using datatable.str.split_into_nhot(): https://datatable.readthedocs.io/en/latest/api/str/split_into_nhot.html

[oleksiyskononenko]

By the way, I've just tried to run your code locally and it works for me

>>> import datatable as dt
>>> import sklearn as sk
>>> DT = dt.Frame(["cat", "dog", "cat", "mouse", "dog"])
>>> preprocess_encoder = sk.preprocessing.OneHotEncoder(sparse=False, handle_unknown='ignore')
>>> preprocess_encoder.fit(DT[0])
OneHotEncoder(handle_unknown='ignore', sparse=False)
>>> encoded = preprocess_encoder.transform(DT[0])
>>> encoded
array([[1., 0., 0.],
       [0., 1., 0.],
       [1., 0., 0.],
       [0., 0., 1.],
       [0., 1., 0.]])

This is on macOS, python 3.9, latest datatable and sklearn. Could you please let me know what is your system and environment?

[oleksiyskononenko]

To get the same result with datatable you just need to do

>>> dt.str.split_into_nhot(DT[0])
   |   cat    dog  mouse
   | bool8  bool8  bool8
-- + -----  -----  -----
 0 |     1      0      0
 1 |     0      1      0
 2 |     1      0      0
 3 |     0      0      1
 4 |     0      1      0
[5 rows x 3 columns]

[Stephane-Thales]

Thank-you all.
After test regarding the sklearn one, it look like I just have to replace mydatatable[:,f['mycol']] by mydatatable[:,'mycol']
Now I will test split_into_nhot, I didn't know it was there.

[oleksiyskononenko]

@Stephane-Thales You can even replace it with mydatatable['mycol'] :)

[Stephane-Thales]

Answer, thanks to @oleksiyskononenko :
Either use mydatatable[:,'mycol'] or mydatatable['mycol'] for sklearn
or use dt.str.split_into_nhot(mydatatable['mycol']) if sklearn is not needed.

[oleksiyskononenko]

Yeah, but mydatatable[:, f['mycol']] should also work as long as you first do

from datatable import f

[Stephane-Thales]

I've the import and it did not work.... not sure why
perhaps an issue with vscode, while using the debug mode by cell... because i rerun with a new kernel and now that work also...
Thanks.