0

So my code like is:

>>> le = preprocessing.LabelEncoder()
>>> le.fit(train["capital city"])
LabelEncoder()
>>> list(le.classes_)
['amsterdam', 'paris', 'tokyo']
>>> le.transform(["tokyo", "tokyo", "paris"])
array([2, 2, 1])
>>> list(le.inverse_transform([2, 2, 1]))
['tokyo', 'tokyo', 'paris']

But what if in my test dataset, I has something like "beijing" but "beijing" does not exist in the training set? Is there a way for the encoder to handle this without adding in every possible capital city in the globe?

Flair
  • 2,609
  • 1
  • 29
  • 41

3 Answers3

2

For a real world scenario, where all you have is training data and new classes can come up later, you can try my solution:

le.classes_ = np.append(le.classes_, "new_class_name")
le.transform(new_y)
Namrata Tolani
  • 823
  • 9
  • 12
  • 2
    This works only when the `new_class_name` would come at the end of all classes, when python does a string sort. Ex: if a,b,d are existing classes, a new class of `e` would work as expected. But if new class = c, this doesn't work. – Dileep Kumar Patchigolla Jan 08 '20 at 11:41
1

You can pass a total list of df['capital city'] to the LabelEncoder.fit() before splitting the dataframe df into train and test.

For example, if df is like this:

df['capital city'] = ['amsterdam', 'paris', 'tokyo', 'beijing', 'tokyo', 'newyork', 'paris']

Then, you can use:

le = preprocessing.LabelEncoder();
le.fit(df['capital city'])

le.classes_
Output: ['amsterdam', 'beijing', 'newyork', 'paris', 'tokyo']

Then use transform() on train and test data to convert them to integers correctly.

train["capital city integers"] = le.transform(train["capital city"])
test["capital city integers"] = le.transform(test["capital city"])

Hope this helps.

Note: Although the above given siggestion will work for you and is perfectly acceptable when you are learning, but you should consider about the real world scenarios when employing this for real tasks. Because in real world, all od your available data will be training data (so you use and encode the capital cities), and then new data may come which contains a never before seen capital city value. What would you like to do in that case?

Vivek Kumar
  • 35,217
  • 8
  • 109
  • 132
  • Uh that's the problem. I am using this in the real world, and I believe your given suggestion isn't scalable. I am hoping this [PR](https://github.com/scikit-learn/scikit-learn/pull/9151) gets merged. Otherwise, I will try to implement my own method in how to handle new categorical information. – Flair Aug 04 '17 at 16:54
  • @Flair . Yes, thats what I asked. Because in real world, if you train your data on one and unseen data comes at prediction time, it may not give good results on i, may fail altogether. – Vivek Kumar Aug 05 '17 at 14:37
0

you can try solution from "sklearn.LabelEncoder with never seen before values" https://stackoverflow.com/a/48169252/9043549

Yury Wallet
  • 1,474
  • 1
  • 13
  • 24