0

I want to use TfidfVectorizer for extracting bigrams. But extending stopwords list does not works with bigrams. How can I fix this problem?

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction import text
import pandas as pd

content = CORPUS
my_stop_words = text.ENGLISH_STOP_WORDS.union(['don know', 'good morning', 'happy birthday'])

vectorizer = TfidfVectorizer(stop_words=my_stop_words, max_features=25, ngram_range=(2, 2))
X = vectorizer.fit_transform(content).todense()
df = pd.DataFrame(X, columns=vectorizer.get_feature_names())
df.to_csv('test.csv')

I got this warning and nothing changed as a result:

Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['birthday', 'don', ...] not in stop_words.
Flodude
  • 255
  • 2
  • 12
  • Does this answer your question? [How to remove stop phrases/stop ngrams (multi-word strings) using pandas/sklearn?](https://stackoverflow.com/questions/45426215/how-to-remove-stop-phrases-stop-ngrams-multi-word-strings-using-pandas-sklearn) – dennlinger Jan 27 '21 at 15:00
  • I'm afraid, I didn't understand that answer! – Flodude Jan 27 '21 at 15:17

0 Answers0