How to calculate F1-micro score using lasagne

Question

import theano.tensor as T
import numpy as np
from nolearn.lasagne import NeuralNet

def multilabel_objective(predictions, targets):
    epsilon = np.float32(1.0e-6)
    one = np.float32(1.0)
    pred = T.clip(predictions, epsilon, one - epsilon)
    return -T.sum(targets * T.log(pred) + (one - targets) * T.log(one - pred), axis=1)

net = NeuralNet(
    # your other parameters here (layers, update, max_epochs...)
    # here are the one you're interested in:
    objective_loss_function=multilabel_objective,
    custom_score=("validation score", lambda x, y: np.mean(np.abs(x - y)))
)

I found this code online and wanted to test it. It did work, the results include training loss, test loss, validation score and during time and so on.

But how can I get the F1-micro score? Also, if I was trying to import scikit-learn to calculate the F1 after adding the following code:

data = data.astype(np.float32) 
classes = classes.astype(np.float32)

net.fit(data, classes)

score = cross_validation.cross_val_score(net, data, classes, scoring='f1', cv=10)

print score

I got this error:

ValueError: Can't handle mix of multilabel-indicator and continuous-multioutput

How to implement F1-micro calculation based on above code?

Are your output continuous (probas between 0 and 1), or binary (either 0 or 1,at least one 1 per row) ? — P. Camilleri, Sep 15 '15 at 09:47
Please avoid including screenshots of code, as they are not searchable. See http://meta.stackoverflow.com/questions/303812/discourage-screenshots-of-code-and-or-errors — P. Camilleri, Sep 15 '15 at 10:05
@M.Massias The results are binary(either 1 or 0) and at least one 1 per low. — Kun, Sep 15 '15 at 14:31
@M.Massias, I asked the related question before and you have assisted me, please check this out:http://stackoverflow.com/questions/32187175/nolearn-for-multi-label-classification, I am confused how to set up the predict value and true value. — Kun, Sep 15 '15 at 14:57
You should already have the true labels if you want to do cross validation (it's part our your data). The predicted labels are obtain using yournet.predict(X_test) where X_test are the test observations. — P. Camilleri, Sep 15 '15 at 15:06
@M.Massias, Can I define the number of layers within NeuralNet function? — Kun, Sep 15 '15 at 16:24
Yes of course, you can add as amny layers as you want. Have you had a look at http://lasagne.readthedocs.org/en/latest/user/tutorial.html ? — P. Camilleri, Sep 15 '15 at 16:26
@M.Massias, thanks for sharing this document. I am using the nonlinearities.sigmoid, performance is good. But I tried to use nonlinearities.softmax, the performance was horrible. — Kun, Sep 15 '15 at 16:35
This is not really related either, but the output of softmax are all between 0 and 1, and sum to 1, whereas it is not true for the sigmoid. So if a point can be in several classes, softmax is not appropriate. I don't know what your background in mathematics, but you might consider reading Bishop's book's (Pattern recognition in machine learning) chapter on neural nets. — P. Camilleri, Sep 15 '15 at 18:13

P. Camilleri · Accepted Answer · 2015-09-15T12:05:59.340

Suppose your true labels on the test set are y_true (shape: (n_samples, n_classes), composed only of 0s and 1s), and your test observations are X_test (shape: (n_samples, n_features)).

Then you get your net predicted values on the test set by y_test = net.predict(X_test).

If you are doing multiclass classification:

Since in your network you have set regression to False, this should be composed of 0s and 1s only, too.

You can compute the micro averaged f1 score with:

from sklearn.metrics import f1_score
f1_score(y_true, y_pred, average='micro')

Small code sample to illustrate this (with dummy data, use your actual y_test and y_true):

from sklearn.metrics import f1_score
import numpy as np


y_true = np.array([[0, 0, 1], [0, 1, 0], [0, 0, 1], [0, 0, 1], [0, 1, 0]])
y_pred = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 0, 1], [0, 0, 1]])

t = f1_score(y_true, y_pred, average='micro')

If you are doing multilabel classification:

You are not outputting a matrix of 0 and 1, but a matrix of probabilities. y_pred[i, j] is the probability that observation i belongs to the class j.

You need to define a threshold value, above which you will say an observation belongs to a given class. Then you can attribute labels accordingly and proceed just the same as in the previous case.

thresh = 0.8  # choose your own value 
y_test_binary = np.where(y_test > thresh, 1, 0) 
# creates an array with 1 where y_test>thresh, 0 elsewhere

f1_score(y_true, y_pred_binary, average='micro')

Yes, I tested the multilabel classification part and it worked. How can I implement the cross validation on this? I used a sample code: pred = net.predict(data[1:50]), thresh = 0.5, y_test_binary = np.where(pred > thresh, 1, 0), t = f1_score(label[1:50],y_test_binary,average='micro') — Kun, Sep 15 '15 at 15:21
From my understanding, I had not implemented the cross validation on calculating the t, right? Or f1_score could do cross validation on its own? — Kun, Sep 15 '15 at 15:28
Then your question is not well posed, because it does not mention cross validation except once in the code. You can compute the f_1 score multiple times, using a different training and test set at each time. Then average these scores — P. Camilleri, Sep 15 '15 at 15:36
I got it and I will try to figure it out. Thank you so much. — Kun, Sep 15 '15 at 15:41

How to calculate F1-micro score using lasagne

1 Answers1

Linked