CountVectorizer: "I" not showing up in vectorized text

Question

I'm new to scikit-learn, and currently studying Naïve Bayes (Multinomial). Right now, I'm working on vectorizing text from sklearn.feature_extraction.text, and for some reason, when I vectorize some text, the word "I" doesn't show up in the outputted array.

Code:

x_train = ['I am a Nigerian hacker', 'I like puppies']

# convert x_train to vectorized text
vectorizer_train = CountVectorizer(min_df=0)
vectorizer_train.fit(x_train)
x_train_array = vectorizer_train.transform(x_train).toarray()

# print vectorized text, feature names
print x_train_array
print vectorizer_train.get_feature_names()

Output:

1 1 0 1 0
0 0 1 0 1
[u'am', u'hacker', u'like', u'nigerian', u'puppies']

Why doesn't "I" seem to show up in the feature names? When I change it to "Ia" or something else like that, it does show up.

score 26 · Accepted Answer · answered Dec 23 '13 at 12:37

This is caused by the default token_pattern for CountVectorizer, which removes tokens of a single character:

>>> vectorizer_train
CountVectorizer(analyzer=u'word', binary=False, charset=None,
        charset_error=None, decode_error=u'strict',
        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
        lowercase=True, max_df=1.0, max_features=None, min_df=0,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern=u'(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)
>>> pattern = re.compile(vectorizer_train.token_pattern, re.UNICODE)
>>> print(pattern.match("I"))
None

To retain "I", use a different pattern, e.g.

>>> vectorizer_train = CountVectorizer(min_df=0, token_pattern=r"\b\w+\b")
>>> vectorizer_train.fit(x_train)
CountVectorizer(analyzer=u'word', binary=False, charset=None,
        charset_error=None, decode_error=u'strict',
        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
        lowercase=True, max_df=1.0, max_features=None, min_df=0,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern='\\b\\w+\\b', tokenizer=None,
        vocabulary=None)
>>> vectorizer_train.get_feature_names()
[u'a', u'am', u'hacker', u'i', u'like', u'nigerian', u'puppies']

Note that the non-informative word "a" is now also retained.

Pattern '(?u)\\b\\w+\\b' is better, because it does not screw words with unicode letters. — usamec, May 04 '15 at 08:35

score 1 · Answer 2 · answered May 21 '15 at 20:36

1

This is because capital letter detection is by default turned off lowercase=True in CountVectorizer

Use

vectorizer_train = CountVectorizer(min_df=0, lowercase=False)

answered May 21 '15 at 20:36

Rizwan Asif

125
1
8

even if lowercase=True, it should have captured the count for 'i'. the accepted answer has the right solution. – Rushdi Shams Nov 09 '17 at 21:24

CountVectorizer: "I" not showing up in vectorized text

2 Answers2

Linked