Retain pandas dataframe structure after SMOTE, oversampling in python

Question

Problem: While implementing SMOTE (a type of oversampling) , my dataframe is getting converted to numpy array).

Test_train_split

from sklearn.model_selection import train_test_split
X_train, X_test, y_train_full, y_test_full = train_test_split(X, y, test_size=0.20, random_state=66)
[IN]type(X_train)
[OUT]pandas.core.frame.DataFrame

After SMOTE, datatype of X_train changes from pandas dataframe to numpy array

from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state = 42)
X_train, y_train = sm.fit_sample(X_train, y_train)
[IN]type(X_train)
[OUT]numpy.ndarray

Expected output I want to retain the dataframe structure of X_train and X_test after SMOTE. How to do that?

Just convert the numpy array back to DataFrame using `pd.DataFrame(data=X_train, columns=X.columns)`. — nishant, Feb 27 '20 at 12:08

noob · Accepted Answer · 2020-02-27T12:12:21.417

12

I found a simpler answer:

from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state = 42)
X_train_oversampled, y_train_oversampled = sm.fit_sample(X_train, y_train)
X_train = pd.DataFrame(X_train_oversampled, columns=X_train.columns)

This helps retain dataframe structure after SMOTE

edited Feb 27 '20 at 12:12

answered Feb 27 '20 at 12:06

noob

3,601
6
27
73

I think it should be `X_train = pd.DataFrame(X_train_oversampled, columns=X_train.columns)`. However, when the DataFrame is created after SMOTE, the data types will not be preserved. – talatccan Feb 27 '20 at 12:09
yes...sorry........this works for me. u can check with any eg – noob Feb 27 '20 at 12:14

score 4 · Answer 2 · answered Feb 27 '20 at 12:02

This function may help you. df is X_train and X_test in your case and output is column name of y as string. SEED is random int in case of if you want to set random_state.

You can use it after split or before split your dataset, depends on your choice.

def smote_sampler(df, output, SEED=33):
     X = df.drop([output], axis=1)
     y = df[output]
     col_names = pd.concat([X, y], axis=1).columns.tolist()
     smt = SMOTE(random_state=SEED)
     X_smote, y_smote = smt.fit_sample(X, y)
     smote_array = np.concatenate([X_smote, y_smote.reshape(-1, 1)], axis=1)
     df_ = pd.DataFrame(smote_array, columns=col_names)
     smote_cols = df_.columns.tolist()
     org_int_cols = df.dtypes.index[df.dtypes == 'int64'].tolist()
     org_float_cols = df.dtypes.index[df.dtypes == 'float64'].tolist()
     try:
         for col in smote_cols:
             if col in org_float_cols:
                 df_[col] = df_[col].astype('float64')
             elif col in org_int_cols:
                 df_[col] = df_[col].astype('int64')
     except:
         raise ValueError
     return df_

Retain pandas dataframe structure after SMOTE, oversampling in python

2 Answers2