CountVectorizer does not print vocabulary

Question

I have installed python 2.7, numpy 1.9.0, scipy 0.15.1 and scikit-learn 0.15.2. Now when I do the following in python:

train_set = ("The sky is blue.", "The sun is bright.")
test_set = ("The sun in the sky is bright.",
"We can see the shining sun, the bright sun.")

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()

print vectorizer


    CountVectorizer(analyzer=u'word', binary=False, charset=None,
    charset_error=None, decode_error=u'strict',
    dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
    lowercase=True, max_df=1.0, max_features=None, min_df=1,
    ngram_range=(1, 1), preprocessor=None, stop_words=None,
    strip_accents=None, token_pattern=u'(?u)\\b\\w\\w+\\b',
    tokenizer=None, vocabulary=None)

     vectorizer.fit_transform(train_set)
    print vectorizer.vocabulary

    None.

Actually it should have printed the following:

CountVectorizer(analyzer__min_n=1,
analyzer__stop_words=set(['all', 'six', 'less', 'being', 'indeed', 'over',    
 'move', 'anyway', 'four', 'not', 'own', 'through', 'yourselves', (...) --->     
For count vectorizer

{'blue': 0, 'sun': 1, 'bright': 2, 'sky': 3} ---> for vocabulary

The above code are from the blog: http://blog.christianperone.com/?p=1589

Could you please help me as to why I get such an error. Since the vocabulary is not indexed properly I am not able to move ahead in understanding the concept of TF-IDF. I am a newbie for python so any help would be appreciated.

Arc.

`vectorizer.vocabulary_` with underscore is what you want. `vectorizer.vocabulary` is not what you want, it's the vocabulary if you passed *in*, if any (usually None). — smci, Feb 19 '20 at 03:15

Balint Domokos · Accepted Answer · 2015-03-06T12:07:57.263

22

You are missing an underscore, try this way:

from sklearn.feature_extraction.text import CountVectorizer
train_set = ("The sky is blue.", "The sun is bright.")
test_set = ("The sun in the sky is bright.", 
    "We can see the shining sun, the bright sun.")

vectorizer = CountVectorizer(stop_words='english')
document_term_matrix = vectorizer.fit_transform(train_set)
print vectorizer.vocabulary_
# {u'blue': 0, u'sun': 3, u'bright': 1, u'sky': 2}

If you use the ipython shell, you can use tab completion, and you can find easier the methods and attributes of objects.

edited Mar 06 '15 at 12:07

answered Mar 06 '15 at 09:00

Balint Domokos

1,021
8
12

Yes. Thanks. I will try using ipython shell so that I do not miss on such tab completion. I have ipython shell never knew this. Thanks for the information. – Archana Mar 06 '15 at 10:46
In the above I also asked how come my stop words=None in CountVectorize which should not be the case. – Archana Mar 06 '15 at 10:52
1

The default value for the stop_words is None. You can create the vectorizer like this: vectorizer = CountVectorizer(stop_words='english') if you want to use the built in english stop words. – Balint Domokos Mar 06 '15 at 12:01
Thanks. I thought the stop words were inbuilt in the function. – Archana Mar 07 '15 at 05:08
One clarification does the vocabulary_ function produce the index terms in alphabetical order. i.e. in the above example 'blue' gets 0 and 1 is given to 'bright' the next term in alphabetical order? – Archana Mar 07 '15 at 10:22
Anyone know why it puts that 'u' in front of the output for vectorizer.vocabulary_? – William Ross Mar 13 '16 at 21:44
@WilliamRoss. `u` in front of a string means strings is in the unicode form. [see here](http://stackoverflow.com/questions/2464959/whats-the-u-prefix-in-a-python-string) – CKM Mar 06 '17 at 08:50

score 5 · Answer 2 · answered Jun 02 '17 at 03:53

Try using the vectorizer.get_feature_names() method. It gives the column names in the order it appears in the document_term_matrix.

from sklearn.feature_extraction.text import CountVectorizer
train_set = ("The sky is blue.", "The sun is bright.")
test_set = ("The sun in the sky is bright.", 
    "We can see the shining sun, the bright sun.")

vectorizer = CountVectorizer(stop_words='english')
document_term_matrix = vectorizer.fit_transform(train_set)
vectorizer.get_feature_names()
#> ['blue', 'bright', 'sky', 'sun']

CountVectorizer does not print vocabulary

2 Answers2

Linked