4

I want to create a custom objective function for training a Keras deep net. I'm researching classification of imbalanced data, and I use the F1 score a lot in scikit-learn. I therefore had the idea of inverting the F1 metric (1 - F1 score) to use it as a loss function/objective for Keras to minimise while training:

(from sklearn.metric import f1_score)

def F1Loss(y_true, y_pred):

    return 1. - f1_score(y_true, y_pred)

However, this f1_score method from scikit-learn requires numpy arrays or lists to calculate the F1 score. I found that Tensors need to be evaluated to their numpy array counterparts using .eval(), which requires a TensorFlow session to perform this task.

I do not know the session object that Keras uses. I have tried using the code below, assuming the Keras backend has its own session object defined somewhere, but this also did not work.

from keras import backend as K

K.eval(y_true)

Admittedly, this was a shot in the dark since I don't really understand the deeper workings of Keras or Tensorflow a the moment.

My question is: how do I evaluate the y_true and y_pred tensors to their numpy array counterparts?

Marcin Możejko
  • 39,542
  • 10
  • 109
  • 120
timbit
  • 353
  • 2
  • 11

2 Answers2

2

Your problem is a classic problem with implementing a discontinous objective in Theano. It's impossible beacuse of two reasons:

  1. F1-score is discontinous : here you can read what should be expected from an objective function in neural networks training. F1-score doesn's satisfy this conditions - so it cannot be used to train neural network.
  2. There is no equivalency between Tensor and Numpy array: it's an fundamental issue. Theano tensor is like x in school equations. You cannot expect from an algebraic variable to be equivalent any object to which it can be assigned to. On the other hand - as a part of a computational graph - a tensor operations should be provided in order to compute objective. If not - you cannot differentiate it w.r.t. parameters what makes most of usual way of training of a neural network impossible.
Community
  • 1
  • 1
Marcin Możejko
  • 39,542
  • 10
  • 109
  • 120
  • This was very helpful! I would not have thought of these points myself. Thank you. I will look for another solution. – timbit Nov 27 '16 at 22:17
0

If you have predicted and actual tensors in numpy array format then I guess that you can use this code snippet:

correct_prediction = tf.equal(tf.argmax(actual_tensor,1), tf.argmax(predicted_tensor,1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

And in keras, I think that you can use this:

model.fit_generator(train_generator, validation_data=val_generator, nb_val_samples=X_val.shape[0],
                        samples_per_epoch=X_train.shape[0], nb_epoch=nb_epoch, verbose=1,
                        callbacks=[model_checkpoint, reduce_lr, tb], max_q_size=1000)

Where train_generator and val_generator generates the training and validation data while training and this also prints the loss and accuracies while training.

Hope this helps...

sansingh
  • 195
  • 8
  • Maybe I'm misunderstanding your answer, but I don't have them in numpy array format. I am actually looking for a way to convert the tensors `y_true` and `y_pred` to numpy arrays. I want them in numpy array format so I can feed them into the F1 function, which simply returns a scalar number. – timbit Nov 25 '16 at 15:46
  • Objects return by .eval() method are in numpy array, check this: http://stackoverflow.com/questions/34097281/how-can-i-convert-a-tensor-into-a-numpy-array-in-tensorflow – sansingh Nov 25 '16 at 15:49
  • Yes, I know. That's why I clearly stated I tried this already in the description of my question. It doesn't work without a session object though and I don't know what session object Keras uses by default. However, as stated by the answer above, using the F1 metric is not a mathematically valid approach anyways, so I'm going to try a different method. – timbit Nov 29 '16 at 10:05