Use pipeline with custom transformer in Scikit Learn

Question

I tried to transform the column 'X' using values in column 'y' (this is a toy example, just to show using y for transformation) before fitted by the last linear regression estimator. But why df['y'] is not passed to MyTransformer?

from sklearn.base import TransformerMixin
class MyTransformer(TransformerMixin):
    def __init__(self):
        pass
    def fit(self, X, y=None):
        return self
    def transform(self, X, y=None):
        print(y)
        return X + np.sum(y)

df = pd.DataFrame(np.array([[2, 3], [1, 5], [1, 1], [5, 6], [1, 2]]), columns=['X', 'y'])
pip =  Pipeline([('my_transformer', MyTransformer()), 
             ('sqrt', FunctionTransformer(np.sqrt, validate=False)),
             ('lr', LinearRegression())])
pip.fit(df[['X']], df['y'])

Running this script will raise an error at line return X + np.sum(y), looks like y is None.

DasHund · Answer 1 · 2019-07-11T00:59:21.653

1

As stated previously, the fit_transform method doesn't pass y off to transform. What I've done previously is implement my own fit_transform. Not your code, but here's an example I wrote recently:

class MultiColumnLabelEncoder:
    def __init__(self, *args, **kwargs):
        self.encoder = StandardLabelEncoder(*args, **kwargs)
    def fit(self, X, y=None):
        return self
    def transform(self,X):
        data = X.copy()
        for i in range(data.shape[1]):
            data[:, i] = LabelEncoder().fit_transform(data[:, i])
        return data
    def fit_transform(self, X, y=None):
        return self.fit(X, y).transform(X)

There are other ways. You could have y as a class param and access it in the transform method.

Edit: I should note that you can pass y off to your version of transform. So:

def fit_transform(self, X, y=None):
    return self.fit(X, y).transform(X, y)

edited Jul 11 '19 at 00:59

answered Jul 11 '19 at 00:53

DasHund

81
4

So basically just use Python's duck typing to bypass TransformerMixin, and everything should be fine, right? – Nicholas Jul 11 '19 at 01:14
You can also extend TransformerMixin, but don't have to. If you do, you get the fit_transform method. The key is really just to override it if you do extend it. Here's the source for that method in sklearn, btw: https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/base.py#L526 – DasHund Jul 11 '19 at 11:21

score 0 · Answer 2 · answered Jul 11 '19 at 00:49

0

The following statement in TransformerMixin will execute ,We can see that transform function only need X parameter

self.fit(X, y, **fit_params).transform(X)

answered Jul 11 '19 at 00:49

wangtianye

306
1
5

Use pipeline with custom transformer in Scikit Learn

2 Answers2

Linked