I have a multi label classification task with FastText. I have to compute the Confusion Matrix for it. I have solved already the problem to compute the CM for a single label. This is the Python script for it:
import argparse
import numpy as np
from sklearn.metrics import confusion_matrix
def parse_labels(path):
with open(path, 'r') as f:
return np.array(list(map(lambda x: x[9:], f.read().split())))
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Display confusion matrix.')
parser.add_argument('test', help='Path to test labels')
parser.add_argument('predict', help='Path to predictions')
args = parser.parse_args()
test_labels = parse_labels(args.test)
print("Test labels:%d (sample)\n%s" % (len(test_labels),test_labels[:1]) )
pred_labels = parse_labels(args.predict)
print("Predicted labels:%d (sample)\n%s" % (len(pred_labels),pred_labels[:1]) )
eq = test_labels == pred_labels
print("Accuracy: " + str(eq.sum() / len(test_labels)))
print(confusion_matrix(test_labels, pred_labels))
This will output something like
Test labels:539328 (sample)
['pop']
Predicted labels:539328 (sample)
['unknown']
Accuracy: 0.17639914857
[[6126 0 0 ..., 0 0 0]
[ 55 0 0 ..., 0 0 0]
[ 6 0 0 ..., 0 0 0]
...,
[ 0 0 0 ..., 0 0 0]
[ 0 0 0 ..., 0 0 0]
[ 0 0 0 ..., 0 0 0]]
The problem is that in the specific case of a multi label task, this is not working properly because I'm computing accuracy
eq = test_labels == pred_labels
eq.sum() / len(test_labels)
that works ok when the files have one column / labels, but not when the predictions output from FastText is a two columns / labels file.
