2

I'm running a very basic code to create encoder classes, and then use the same classes to encode a new dataframe. In this code, I don't need to use np.save and np.load, however in my actual implementation, I will need to re-load the encoder to transform a new dataframe. I'm trying to understand how I can create an encoder class on one dataframe, and then in another script, load that encoder and transform a new dataframe.

from sklearn.preprocessing import LabelEncoder
import pickle as cPickle
import numpy as np

df_test = pd.DataFrame({'A': [1, 2, 3, 4],
                   'B': ["Yes", "No", "Yes", "Yes"],
                   'C': ["Yes", "No", "No", "Yes"],
                   'D': ["No", "Yes", "No", "Yes"]})

le = LabelEncoder()
df_test['A'] = le.fit_transform(df_test['A'])
le_dict = dict(zip(le.classes_, le.transform(le.classes_)))
class_name = 'classes_' +'A' + '.npy'
np.save(class_name, le_dict, allow_pickle=True)

print(df_test)
print(le.classes_)

le.classes_ = np.load('classes_A.npy', allow_pickle = True)

print(le.classes_)

df_new = pd.DataFrame({'A': [1, 2, 3, 4],
                   'B': ["Yes", "No", "Yes", "Yes"],
                   'C': ["Yes", "No", "No", "Yes"],
                   'D': ["No", "Yes", "No", "Yes"]})

df_new['A'] = le.transform(df_new['A'])

This is giving me the following error:


  File "<ipython-input-42-a1aa630ec7e8>", line 1, in <module>
    df_new['A'] = le.transform(df_new['A'])

  File "/Users/usr/opt/anaconda3/envs/signals_gcp_py36/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 257, in transform
    _, y = _encode(y, uniques=self.classes_, encode=True)

  File "/Users/usr/opt/anaconda3/envs/signals_gcp_py36/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 110, in _encode
    return _encode_numpy(values, uniques, encode)

  File "/Users/usr/opt/anaconda3/envs/signals_gcp_py36/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 49, in _encode_numpy
    % str(diff))

ValueError: y contains previously unseen labels: [1, 2, 3, 4]

When I print le.classes_ before loading it from memory, this is what it looks like: array([1, 2, 3, 4])

But when I print it after np.load(), this is what it looks like: {1: 0, 2: 1, 3: 2, 4: 3}

Here's some more information on le.classes after np.load():

In []: le.classes_
Out[]: array({1: 0, 2: 1, 3: 2, 4: 3}, dtype=object)

In []: type(le.classes_)
Out[]: numpy.ndarray

In []: print(le.classes_)
Out[]: {1: 0, 2: 1, 3: 2, 4: 3}

I'm trying to understand what is happening how these functions work. I ran the same code, but for col.B, and I'm getting yet another error.

from sklearn.preprocessing import LabelEncoder
import pickle as cPickle
import numpy as np

df_test = pd.DataFrame({'A': [1, 2, 3, 4],
                   'B': ["Yes", "No", "Yes", "Yes"],
                   'C': ["Yes", "No", "No", "Yes"],
                   'D': ["No", "Yes", "No", "Yes"]})

le = LabelEncoder()
df_test['B'] = le.fit_transform(df_test['B'])
le_dict = dict(zip(le.classes_, le.transform(le.classes_)))
class_name = 'classes_' +'B' + '.npy'

np.save(class_name, le_dict, allow_pickle=True)

print(df_test)
print(le.classes_)

le.classes_ = np.load('classes_B.npy', allow_pickle = True)

print(le.classes_)

df_new = pd.DataFrame({'A': [1, 2, 3, 4],
                   'B': ["Yes", "No", "Yes", "Yes"],
                   'C': ["Yes", "No", "No", "Yes"],
                   'D': ["No", "Yes", "No", "Yes"]})

df_new['B'] = le.transform(df_new['B'])

The error with this one is TypeError: argument must be a string or number.

Here's the full stack:

Traceback (most recent call last):

  File "<ipython-input-71-a0243d411c34>", line 1, in <module>
    df_new['B'] = le.transform(df_new['B'])

  File "/Users/usr/opt/anaconda3/envs/signals_gcp_py36/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 257, in transform
    _, y = _encode(y, uniques=self.classes_, encode=True)

  File "/Users/usr/opt/anaconda3/envs/signals_gcp_py36/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 107, in _encode
    raise TypeError("argument must be a string or number")

TypeError: argument must be a string or number

Any help is appreciated!

Jo Bennet
  • 131
  • 6

0 Answers0