Same accuracy and F1 score while doing multi label classification

Question

I have written a code based on this site and made different multi-label classifiers.

I would like to evaluate my model based on accuracy per class and F1 measurement per class.

The problem is that I am getting the same number for both accuracy and f1 measurement in all models.

I am suspicious I have done something wrong. I would like to know in which circumstances this may happen.

the code is exactly the same as the site and I calculated the f1 measurement like this:

print('Logistic Test accuracy is {} '.format(accuracy_score(test[category], prediction)))
    print 'Logistic f1 measurement is {} '.format(f1_score(test[category], prediction, average='micro'))

Update 1

this is the whole code,

df = pd.read_csv("finalupdatedothers.csv")
categories = ['ADR','WD','EF','INF','SSI','DI','others']

train,test = train_test_split(df,random_state=42,test_size=0.3,shuffle=True)
X_train = train.sentences
X_test = test.sentences

NB_pipeline = Pipeline([('tfidf', TfidfVectorizer(stop_words=stop_words)),
                        ('clf',OneVsRestClassifier(MultinomialNB(fit_prior=True,class_prior=None))),])
for category in categories:
    print 'processing {} '.format(category)
    NB_pipeline.fit(X_train,train[category])
    prediction = NB_pipeline.predict(X_test)
    print 'NB test accuracy is {} '.format(accuracy_score(test[category],prediction))
    print 'NB f1 measurement is {} '.format(f1_score(test[category],prediction,average='micro'))
    print "\n"

and this is the output:

processing ADR 
NB test accuracy is 0.821963394343 
NB f1 measurement is 0.821963394343 

and this is the way my data looks:

,sentences,ADR,WD,EF,INF,SSI,DI,others
0,"extreme weight gain, short-term memory loss, hair loss.",1,0,0,0,0,0,0
1,I am detoxing from Lexapro now.,0,0,0,0,0,0,1
2,I slowly cut my dosage over several months and took vitamin supplements to help.,0,0,0,0,0,0,1
3,I am now 10 days completely off and OMG is it rough.,0,0,0,0,0,0,1
4,"I have flu-like symptoms, dizziness, major mood swings, lots of anxiety, tiredness.",0,1,0,0,0,0,0
5,I have no idea when this will end.,1,0,0,0,0,0,1

Why am I getting the same number?

Thanks.

Answer 1

By doing this:

for category in categories:
...
...

You are essentially turning the problem from multi-label to binary. If you want to proceed with this, then no need for the OneVsRestClassifier. You can use the MultinomialNB directly. Or else you can directly do this with OneVsRestClassifier:

# Send all labels at once.
NB_pipeline.fit(X_train,train[categories])
prediction = NB_pipeline.predict(X_test)
print 'NB test accuracy is {} '.format(accuracy_score(test[categories],prediction))
print 'NB f1 measurement is {} '.format(f1_score(test[categories],prediction, average='micro'))

It may throw some warnings about some labels present in all the training data, but thats because the sample data you posted is too small.

@user2906838, you are correct about scores. When average='micro', the results produced will be equal. This is mentioned in documentation here:

Note that for “micro”-averaging in a multiclass setting with all labels included will produce equal precision, recall and F,

Its written about multi-class there, but I doubt that its the same for binary also. See this similar question in which the user has calculated all the scores manually: Multi-class Clasification (multiclassification): Micro-Average Accuracy, Precision, Recall and F Score All Equal

Answer 2

Well, it's probably because both accuracy_score and f1_score are returning the same score. Although there discrepancies between how they are calculated, the result. If you want to know more about how they are calculated, there is already an answer for it here: How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?

Regarding your current problem of the same score, Please change the value of average from micro to weighted. This should essentially change your scores. As I pointed in the comment.