2

I am trying to implement the cross entropy loss between two images for a fully conv Net. I have both my training and input images in the range 0-1. Now, I am trying to implement this for only one class of images. To illustrate say I have different orange pictures but only orange pictures. I've built my model and I have implemented a cross entropy loss function. 

def loss_func_entropy(logits,y):
    logits=tf.reshape(logits,[BATCH_SIZE*480*640,1])
    y=tf.reshape(y,[BATCH_SIZE*480*640,1])
    print (logits.get_shape(),y.get_shape())
    return tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits,labels=y,dim=0) )

Obviously I am not doing this right because my loss function keeps increasing. Things to note is that logits and y are both 2D. i reshape them into a single vector and try to do cross entropy.

kauDaOtha
  • 1,140
  • 1
  • 14
  • 23

1 Answers1

5

First, the reshaping should actually be (it will run without it too for this particular problem, but is not really what TF would expect to see as the output of the model):

logits=tf.reshape(logits,[BATCH_SIZE,480*640])
y=tf.reshape(y,[BATCH_SIZE,480*640])

Then the only error lies in the fact that TF has a "jungle" of cross-entropy functions. The one you are looking for is sigmoid_cross_entropy_with_logits, not the softmax one. The one you used normalises the whole image, so that sum of pixels is one (which obviously is not true). What you want is to treat each pixel as a separate "soft classification" problem, where its normalised intensity is considered a probability and this is exactly what sigmoid_cross_entropy_with_logits does. In other words - this is just a multi-label classification with soft targets.

In particular, with the previous reshaping, softmax_cross_entropy_with_logits would behave even more oddly - since you had one output, yet still applied softmax, it should always produce output "1", as for a scalar x:

softmax(x) = exp(x) / SUM exp(x) = 1
lejlot
  • 64,777
  • 8
  • 131
  • 164
  • Thanks. I now am able to get a decreasing loss function. Could you elaborate on why I should choose sigmoid cross entropy. If you could point out to any resources behind the theory it would be useful. – kauDaOtha Aug 27 '17 at 23:25
  • I elaborated a lot in the linked post about jungle of cross entropies. This is just about subtle differences in setting that have a separate helper function. For your case this boils down to what is described here - softmax_* is used when targets sum to 1 (in your case - all pixels together), while sigmoid_* when **each dimension** of the target is in [0,1] (so **every** pixel independently). – lejlot Aug 27 '17 at 23:28
  • Makes sense . Thanks a ton! – kauDaOtha Aug 27 '17 at 23:30
  • One more thing is that "mathematically" the shape should be `tf.reshape(logits,[BATCH_SIZE,480*640])`. This will not change how the code works, but this is how the problem is usually described. – lejlot Aug 27 '17 at 23:32