pandas categories new levels

Question

How does pandas categorical https://pandas.pydata.org/pandas-docs/stable/categorical.html handle new and unseen levels? I am thinking about a scikit-learn like setup. Currently, I have something like: https://gist.github.com/geoHeil/5caff5236b4850d673b2c9b0799dc2ce

def: fit() 
  for each column:
   fit a label encoder:
def: transform()
  for each column:
     check if column was unseen
       yes(unseen) replace 
       no: label encode

but this is pretty slow.

Apparently, decision trees like xgboost or lightbm can directly handle categorical data, i.e. one would not need to fiddle around manually with this slow conversion. But when looking at their code https://github.com/Microsoft/LightGBM/blob/master/python-package/lightgbm/sklearn.py#L532 they seem to use LGBMLabelEncoderwhich is a standard scikit-learn LabelEncoder.

I wonder how that can handle unseen data.

If a manual conversion is required would pandas.Categorical allow a quicker conversion - even if unseen levels are in the new data?

edit

Please see https://github.com/geoHeil/pythonQuestions/blob/master/categorical-encoding.ipynb for an overview how I could not get scikit-learn's usual suspects to work. Still looking for something more performant than my solution. Also lightGBM https://github.com/Microsoft/LightGBM/issues/789 suggests to use custom encoding strategy.

Answer 1

There might be a pandas solutin, but it works probably best with sklearns LabelBinarizer

from sklearn.preprocessing import LabelBinarizer
df= pd.DataFrame({'A':['a','b','c','a']})
lb = LabelBinarizer()
lb.fit(df["A"])
lb.transform(df["A"])

[[1 0 0]
 [0 1 0]
 [0 0 1]
 [1 0 0]]

df2 = pd.DataFrame({'A':['a','b','d']})
lb.transform(df2['A'])
[[1 0 0]
 [0 1 0]
 [0 0 0]]

So we see that 'd' is essentially mapped to neither 'a','b' or 'c'. Note however, that there is a bug which probably will be resolved in one of the next sklearn releases.

The LabelBinarizer is fit during training and recalls the values passed to it. New values get mapped to all zeros. It might be more feasible do write a transformer (as seen here before the edit) using pandas get_dummies.

This could be quite straightforward due to name matching of columns. Fit in the first step and store the column names, than just transform in the transformstep, but only keep column names that you identified in fitting (potentially adding zome zero columns if training levels are not present in the test set). Then you are done ;)