How to get feature Importance in naive bayes?

Question

I have a dataset of reviews which has a class label of positive/negative. I am applying Naive Bayes to that reviews dataset. Firstly, I am converting into Bag of words. Here sorted_data['Text'] is reviews and final_counts is a sparse matrix

count_vect = CountVectorizer() 
final_counts = count_vect.fit_transform(sorted_data['Text'].values)

I am splitting the data into train and test dataset.

X_1, X_test, y_1, y_test = cross_validation.train_test_split(final_counts, labels, test_size=0.3, random_state=0)

I am applying the naive bayes algorithm as follows

optimal_alpha = 1
NB_optimal = BernoulliNB(alpha=optimal_aplha)

# fitting the model
NB_optimal.fit(X_tr, y_tr)

# predict the response
pred = NB_optimal.predict(X_test)

# evaluate accuracy
acc = accuracy_score(y_test, pred) * 100
print('\nThe accuracy of the NB classifier for k = %d is %f%%' % (optimal_aplha, acc))

Here X_test is test dataset in which pred variable gives us whether the vector in X_test is positive or negative class.

The X_test shape is (54626 rows, 82343 dimensions)

length of pred is 54626

My question is I want to get the words with highest probability in each vector so that I can get to know by the words that why it predicted as positive or negative class. Therefore, how to get the words which have highest probability in each vector?

score 24 · Accepted Answer · edited Nov 14 '22 at 19:23

24

You can get the importantance of each word out of the fit model by using the coefs_ or feature_log_prob_ attributes. For example

neg_class_prob_sorted = NB_optimal.feature_log_prob_[0, :].argsort()[::-1]
pos_class_prob_sorted = NB_optimal.feature_log_prob_[1, :].argsort()[::-1]

print(np.take(count_vect.get_feature_names(), neg_class_prob_sorted[:10]))
print(np.take(count_vect.get_feature_names(), pos_class_prob_sorted[:10]))

Prints the top 10 most predictive words for each of your classes.

edited Nov 14 '22 at 19:23

wordsforthewise

13,746
5
87
117

answered May 25 '18 at 13:47

piman314

5,285
23
35

2

Thanks a lot. It was so helpful. You saved me. – merkle May 26 '18 at 03:10
13

I think `np.take(count_vect.get_feature_names(), neg_class_prob_sorted[:10])` returns the 10 least important features. – Yuri Malheiros Feb 20 '19 at 11:33
@piman314 print(np.take(count_vect.get_feature_names(), neg_class_prob_sorted[:10])) belongs to which class ? also, can you please tell me, along with the feature names how can I print its corresponding probability value? – Atif Sheikh Dec 06 '19 at 17:38
The `argsort` output needs to be reversed to get the most important features: `neg_class_prob_sorted = NB_optimal.feature_log_prob_[0, :].argsort()[::-1]` – Emaad Ahmed Manzoor Jun 19 '20 at 19:16
The answer from @dimid is correct as well. – VeilEclipse Mar 17 '21 at 01:39

Tom · Answer 2 · 2023-06-06T04:09:59.897

5

I had the same trouble, maybe this is for datascience exchange forum but I want to post it here since I achieved a very good result.

First:

Stands for positive class ,

Stands for negative class. P() stands for proability.

We are going to build odds ratio, which can be demostrated that it is equal to P(word i ,+) / P(word i ,-) (let me know if you need the demostration of it guys). If this ratio is greater than 1 means that the word i is more likely to occur in a positive texts than in negative text.

These are the priors in the naive bayes model:

prob_pos = df_train['y'].value_counts()[0]/len(df_train)
prob_neg = df_train['y'].value_counts()[1]/len(df_train)

Create a dataframe for storing the words

df_nbf = pd.DataFrame()
df_nbf.index = count_vect.get_feature_names()
# Convert log probabilities to probabilities. 
df_nbf['pos'] = np.e**(nb.feature_log_prob_[0, :])
df_nbf['neg'] = np.e**(nb.feature_log_prob_[1, :])

 
df_nbf['odds_positive'] = (nb.feature_log_prob_[0, :])/(nb.feature_log_prob_[1, :])*(prob_pos /prob_neg)

df_nbf['odds_negative'] = (nb.feature_log_prob_[1, :])/(nb.feature_log_prob_[0, :])*(prob_neg/prob_pos )

Most important words. This will hive you a >1 ratio. For example a odds_ratio_negative =2 for the word "damn" means that this word is twice likely to occur when the comment or your class is negative in comparison with your positive class.

# Here are the top5 most important words of your positive class:
odds_pos_top5 = df_nbf.sort_values('odds_positive',ascending=False)['odds_positive'][:5]
# Here are the top5 most important words of your negative class:
odds_neg_top5 = df_nbf.sort_values('odds_negative',ascending=False)['odds_negative'][:5]

edited Jun 06 '23 at 04:09

answered Jun 03 '20 at 14:19

Tom

496
8
16

1

The other answers does not give you var importance since this is the log of the Prob( word / + ) for example. And for example the word "the " has a very high probability for positive class as well as for negative class. Therefore both in the naive bayes weight the same. and could be treated as a constant ( it means it does not change the prob of being of one class or another) – Tom Jun 03 '20 at 14:35
1

Thanks, an interesting approach. Shouldn't we use e instead of 10 for exponentiation in `df_nbf['pos']`? – dimid Jun 19 '20 at 08:32
1

Hi, you are raising a very valid point. I have read sklearn documentation and they don't say anything about what log transformation they are using. I've got very interesting results using 10 exponenciation and assumed that sklearn were using log10 transformation for the probabilities. Please let me know if I am wrong. To clarify to the readers the order of the var importance is unlikely to change, but the interpretation that I have stated would change if I am not using the right transformation. – Tom Jun 19 '20 at 14:10
1

Looking at the source code I saw `np.lop` and not `np.log10`. – dimid Jun 19 '20 at 14:42
2

Very late to the party, but I think many people are still looking for this: This answer, although not that beautiful, really is the one on point. When people ask about important features in a binary classification task, they typically want the features most predictive of the class. All the other answers (and all others I have seen elsewhere) don't actually give you the those most informative features. For instance, features appearing in all documents should be uninformative, but will have a high feature importance, according to the other answers – JBN Apr 10 '21 at 06:04
1

@JBN , yeah, according to sklearn documentation, feature_log_prob_ndarray is just ln(Prob(xi / class=k ) , so the fact that a word is present very present in one class does not imply that it wion't be present in the other class, what matters after all is the words that are more present in one class instead of the other one – Tom Apr 13 '21 at 17:54
@Tom - This is actually a great way to explain word importance. Could you please enlighten on how this calculations would change if you had three classes or more than two classes? Will the denominator change like `df_nbf['odds_positive'] = (nb.feature_log_prob_[0, :])/(nb.feature_log_prob_[1, :] + nb.feature_log_prob[2,:])*(prob_0/(prob_1+prob_2))` ? Also, why are you multiplying the ratio of class probabilities where you already individual class probabilities i.e. `prob_pos` and `prob_neg`? – Krishnang K Dalal Jun 05 '23 at 03:42

score 4 · Answer 3 · answered Jun 19 '20 at 08:17

def get_salient_words(nb_clf, vect, class_ind):
    """Return salient words for given class
    Parameters
    ----------
    nb_clf : a Naive Bayes classifier (e.g. MultinomialNB, BernoulliNB)
    vect : CountVectorizer
    class_ind : int
    Returns
    -------
    list
        a sorted list of (word, log prob) sorted by log probability in descending order.
    """

    words = vect.get_feature_names()
    zipped = list(zip(words, nb_clf.feature_log_prob_[class_ind]))
    sorted_zip = sorted(zipped, key=lambda t: t[1], reverse=True)

    return sorted_zip

neg_salient_top_20 = get_salient_words(NB_optimal, count_vect, 0)[:20]
pos_salient_top_20 = get_salient_words(NB_optimal, count_vect, 1)[:20]

score 2 · Answer 4 · answered May 25 '18 at 12:17

2

Try this:

pred_proba = NB_optimal.predict_proba(X_test)
words = np.take(count_vect.get_feature_names(), pred_proba.argmax(axis=1))

answered May 25 '18 at 12:17

MaxU - stand with Ukraine

205,989
36
386
419

It is not clear how your answer will return the most important features as per the classifier. Your code selects the feature names with indices that correspond to the class with the highest probability for each test input, i.e. indices from [0, n_classes-1], and those indices need not be related to the most important features at all. Also, it should be possible to determine the importance of various features right after training (fit / fit_transform) and shouldn't need the test data at all. Shouldn't the indices be chosen based on feature_log_prob_ as mentioned in other answers? – so2 Dec 21 '21 at 16:35

How to get feature Importance in naive bayes?

4 Answers4

Linked