How to implement Weighted Binary CrossEntropy on theano?

Question

How to implement Weighted Binary CrossEntropy on theano?

My Convolutional neural network only predict 0 ~~ 1 (sigmoid).

I want to penalize my predictions in this way :

Basically, i want to penalize MORE when the model predicts 0 but the truth was 1.

Question : How can I create this Weighted Binary CrossEntropy function using theano and lasagne ?

I tried this below

prediction = lasagne.layers.get_output(model)


import theano.tensor as T
def weighted_crossentropy(predictions, targets):

    # Copy the tensor
    tgt = targets.copy("tgt")

    # Make it a vector
    # tgt = tgt.flatten()
    # tgt = tgt.reshape(3000)
    # tgt = tgt.dimshuffle(1,0)

    newshape = (T.shape(tgt)[0])
    tgt = T.reshape(tgt, newshape)

   #Process it so [index] < 0.5 = 0 , and [index] >= 0.5 = 1


    # Make it an integer.
    tgt = T.cast(tgt, 'int32')


    weights_per_label = theano.shared(lasagne.utils.floatX([0.2, 0.4]))

    weights = weights_per_label[tgt]  # returns a targets-shaped weight matrix
    loss = lasagne.objectives.aggregate(T.nnet.binary_crossentropy(predictions, tgt), weights=weights)

    return loss

loss_or_grads = weighted_crossentropy(prediction, self.target_var)

But I get this error below :

TypeError: New shape in reshape must be a vector or a list/tuple of scalar. Got Subtensor{int64}.0 after conversion to a vector.

Reference : https://github.com/fchollet/keras/issues/2115

Reference : https://groups.google.com/forum/#!topic/theano-users/R_Q4uG9BXp8

Since you are using `binary_crossentropy(..)` there will be a penalty for every wrong prediction. What you actually need is to look at how to deal with imbalanced datasets, this [link](http://www.svds.com/learning-imbalanced-classes/) might be helpful. — uyaseen, Sep 09 '16 at 13:32

score 2 · Accepted Answer · answered Dec 20 '16 at 19:33

Thanks to the developers on lasagne group, i fixed this by constructing my own loss function.

loss_or_grads = -(customized_rate * target_var * tensor.log(prediction) + (1.0 - target_var) * tensor.log(1.0 - prediction))

loss_or_grads = loss_or_grads.mean()

score 0 · Answer 2 · edited May 23 '17 at 11:45

0

To address your syntax error:

Change

newshape = (T.shape(tgt)[0])
tgt = T.reshape(tgt, newshape)

to

newshape = (T.shape(tgt)[0],)
tgt = T.reshape(tgt, newshape)

T.reshape expects a tuple of axes, you didn't provide this, hence the error.

Before penalizing false-negatives (prediction 0, truth 1) make sure that this prediction error is not based on the statistics of your training data, as @uyaseen suggested.

edited May 23 '17 at 11:45

Community

1
1

answered Sep 16 '16 at 17:00

nemo

55,207
13
135
135

But how can i penalize more certain targets/predictions ? Like on the table/image i posted? – KenobiBastila Sep 17 '16 at 14:26
You can use class weights to boost the loss of the class you care most about. This is mostly used for balancing training data but it is generally just a way to boost the errors from one particular class. – nemo Sep 17 '16 at 19:30
Yeah but how i implement on lasagne? – KenobiBastila Sep 17 '16 at 19:53

How to implement Weighted Binary CrossEntropy on theano?

2 Answers2

Linked