Text and dummy variables in ML - features selection

Question

I have a dataframe like this:

       Text                                             A   B   C    Label
337 nobodi can explain gave what we did ...             0   1   0      1
338 provide an example                                  1   1   0      0
339 another one????                                     1   0   0      1

I would like to understand how to build a ML classifier. Currently, I did as follows:

X = train[['Text','A','B','C']]
y = train['Label']

# Split into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=40)
X_train, X_valid, y_train, y_valid  = train_test_split(X, y, test_size=0.25, random_state=40) 
# Returning to one dataframe
train_df = pd.concat([X_train, y_train], axis=1)
test_df = pd.concat([X_test, y_test], axis=1)
valid_df = pd.concat([X_valid, y_valid], axis=1)

Then I create features using BOW and TFIDF:

countV = CountVectorizer()
train_count = countV.fit_transform(train_df['Text'].values)

# To create tfidf frequency features

tfidfV = TfidfTransformer()
train_tfidf = tfidfV.fit_transform(train_count)

tfidf_ngram = TfidfVectorizer(stop_words='english',ngram_range=. (1,2),use_idf=True,smooth_idf=True)

However, when I build the models, for example a NB model:

nb_pipeline = Pipeline([
        ('NBCV', countV),
        ('nb_clf',MultinomialNB())])

nb_pipeline.fit(train_df['Text'],train_df['Label'])
predicted_nb = nb_pipeline.predict(test_df['Text'])
np.mean(predicted_nb == test_df['Label'])

Something does not work, as I loose information on my dummy variables A,B,C. I have only features from Text. I can check this when I try to look at features importance:

feature_names = nb_pipeline.named_steps["NBCV"].get_feature_names()
coefs = nb_pipeline.named_steps["nb_clf"].coef_.flatten()

import pandas as pd
zipped = zip(feature_names, coefs)
df = pd.DataFrame(zipped, columns=["feature", "value"])
df["ABS"] = df["value"].apply(lambda x: abs(x))
df["colors"] = df["value"].apply(lambda x: "green" if x > 0 else "red")
df = df.sort_values("ABS", ascending=True)

Can you explain me why I am loosing this information and how I can keep my dummy variables in the model? Those variables should be very meaningful for the model, so I cannot exclude them from the model build. I need to check accuracy of the model and see the impact of those variables on that.

Answer 1

There are a few things to unpack here, so let's walk through them:

Validation, Test, Training data

Firstly, never mix your validation and testing data. This article provides more than 5 different quotes from various academic textbooks, including leading industry leaders such as Max Kuhn, that specifically highlight why you need to provide the model with completely untouched and unseen data for final evaluation. I would suggest reading that.

Getting the pipeline to work properly

In order to get your pipeline to work properly, you will need to use [make_column_transformer][3], which is new as of sklearn v 20.0.0. I tried to recycle as much code as I could from your example, so please note any subtle differences.

#!/usr/bin/env python
# coding: utf-8

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB

# Added
from sklearn.compose import make_column_transformer

# Create some test data to mimic OP's example

bad_sentences = ["I hated this restauraunt!", "I do not like Stack Overflow", "I am sad about the economy", 
                 "I do not feel good about the new Python update", "I think I am allergic to hazelnut!", "I will never eat here",
                 "I am never coming back!", "I feel really sad after hearing the news", "George is getting upset!", "Python!"]
good_sentences = ["I really liked this place", "Stack Overflow is a great resource", "I am glad we are getting vaccinated",
                  "I love anime", "I miss you too!", "I am upset we don't hang out more, I love it!", 
                  "Why are we feeling sad? Be happy!", "Hi, my name is wundermahn, what is your name?", "Lol!", 
                  "I am the master of my domain"]

# Create things a bit more verbosely so OP understands what we are doing

sentences = bad_sentences+good_sentences

a_bad = [1 for i in range(len(bad_sentences))]
a_good = [0 for i in range(len(good_sentences))]
a = a_bad+a_good

b_bad = [1 for i in range(len(bad_sentences))]
b_good = [0 for i in range(len(good_sentences))]
b = b_bad+b_good

c_bad = [1 for i in range(len(bad_sentences))]
c_good = [0 for i in range(len(good_sentences))]
c = c_bad+c_good

label_bad = [1 for i in range(len(bad_sentences))]
label_good = [0 for i in range(len(good_sentences))]
label = label_bad+label_good

# Create dataframe
df = pd.DataFrame({'Text': sentences, 'A': a, 'B': b, 'C': c, 'label': label})

# NEVER mix your validation and testing data!
# https://stackoverflow.com/questions/28556942/pandas-remove-rows-at-random-without-shuffling-dataset

np.random.seed(38)
remove_n = 2
drop_indices = np.random.choice(df.index, remove_n, replace=False)
valid_df = df.iloc[drop_indices]
remaining_df = df.drop(drop_indices)

X_train, X_test, y_train, y_test = train_test_split(remaining_df[[col for col in remaining_df.columns if col != 'label']],
                                                    remaining_df['label'], test_size=0.2, random_state=38)

# Get CountVectorizer working as an example, you can add tfidf later on

# Now, create your pipeline which should include your vectorizer, as well as your model you plan on training
nb_pipeline = Pipeline([
                        ('vectorizer', make_column_transformer((CountVectorizer(), 'Text'), remainder='passthrough')),
                        ('classifier', MultinomialNB())
                      ])

# Now, we can effectively train our model using the proper feature set
nb_pipeline.fit(X_train, y_train)


# Now, get prediction
predicted_nb = nb_pipeline.predict(X_test)

# Print accuracy
print(np.mean(predicted_nb==y_test))

# Get feature names

# Note, we need to slightly edit how we get the names now that we are using a different transformation pipeline
# https://stackoverflow.com/questions/54646709/sklearn-pipeline-get-feature-names-after-onehotencode-in-columntransformer
feature_names = nb_pipeline['vectorizer'].transformers_[0][1].get_feature_names()
coefs = nb_pipeline.named_steps["classifier"].coef_.flatten()

# Your code
zipped = zip(feature_names, coefs)
features_df = pd.DataFrame(zipped, columns=["feature", "value"])
features_df["ABS"] = features_df["value"].apply(lambda x: abs(x))
features_df["colors"] = features_df["value"].apply(lambda x: "green" if x > 0 else "red")
features_df = df.sort_values("ABS", ascending=True)

# See results on validation set
valid_preds = nb_pipeline.predict(valid_df[['Text', 'A', 'B', 'C']])
print(np.mean(valid_preds==valid_df['label']))

Please note, I am using: Python 3.8.8, sklearn 0.24.1, pandas 1.2.4, numpy 1.20.2