3

I am writing a custom loss function and I want to take 1 - logarithm of the value calculated. I am writing in keras with tensorflow backend as follows:

nonlabels = list(np.where(predictions<threshold)[1])
loss_fg_nT = 0
probs_nT = tf.gather(probs_fg,nonlabels,axis=3)
for i in range(len(nonlabels)):
    probs_temp = tf.reshape(probs_nT[:,:,:,i],[-1])
    prob_max = tf.math.reduce_max(probs_temp)
    const = tf.constant(0.000001)
    prob_max = tf.math.add(prob_max,const)
    #prob_max = tf.math.subtract(tf.constant(1.0),prob_max)
    val = K.log(prob_max)
    loss_fg_nT -= val
    loss_fg_nT = loss_fg_nT/(len(nonlabels)+0.000001)

If i remove prob_max = tf.math.subtract(tf.constant(1.0),prob_max) line the function is working fine but when I take log of (1-prob_max) value it is giving nan value. I couldn't understand the reason behind this as I am adding a nominal value to avoid taking log 0 as well. Can someone help me correct this error.

OverLordGoldDragon
  • 1
  • 9
  • 53
  • 101
Sree
  • 973
  • 2
  • 14
  • 32
  • For which value of `prob_max` does `K.log` yield nans? You can find out via a `try-except` block; also consider using `K.epsilon()` instead of the `0.00..1`, make sure all `reduce_max` outputs are positive, and try `tf.math.log` instead of `K.log` – OverLordGoldDragon Nov 06 '19 at 03:48
  • it is giving `nan` for every value. I used `K.epsilon()` and also did `tf.math.abs(probs_max)` just to avoid negative values and used `tf.math.log`. But still I am facing the same issue. This is so weird!! – Sree Nov 06 '19 at 04:11
  • Yes, it is odd - too odd; possible install issue. Before going there, what is the output of `print(K.get_value(K.log(tf.constant(0.9)))`? Let me know if `nan`, it's a broken install I can suggest steps for. Lastly, where are you importing your `keras.backend` from? Make sure it's consistent with other imports, and that it is **not** [`tf.python.keras`](https://stackoverflow.com/questions/58279628/what-is-the-difference-between-tf-keras-and-tf-python-keras) – OverLordGoldDragon Nov 06 '19 at 13:37
  • @OverLordGoldDragon, `print(K.get_value(K.log(tf.constant(0.9)))`, this is giving me proper value not `nan`. I imported keras backend as `keras.backend`. – Sree Nov 06 '19 at 16:56
  • I'd find the output of [this script](https://pastebin.com/z8mrS82n) helpful; you can paste it here as pastebin – OverLordGoldDragon Nov 06 '19 at 16:58

2 Answers2

4

You are getting nan error because you K.log() for a minus value. Additionally, you get inf error if you K.log() for a zero value. I attach an image to demonstrate that attach_1

I also attach the code that I used to debug, hope this help you can manually debug in the future attach_2

Happy coding!

Abhilash Chandran
  • 6,803
  • 3
  • 32
  • 50
  • But even if I did `tf.math.abs(probs_max)`, I am getting the same error. this ensures that I always have a positive value right? Correct me if I am wrong. – Sree Nov 06 '19 at 04:31
  • Yes, using tf.math.abs() makes the value positive, my "attach_2" in last step becames ~13,8023 instead of NaN too, please try to use alternative names for prob_max like I did, I once got error because I keep same name of prob_max, so I change to prob_max_1 , prob_max _2, prob_max _3, and please make sure the dimension of prob_max is correct (1,) and range from (0-1) –  Nov 06 '19 at 04:37
  • The value of prob_max is definitely between 0 and 1 as I have used softmax function to get the probabilities. I still have the issue after renaming the variables. – Sree Nov 06 '19 at 05:17
  • Could you print out the value of this line prob_max = tf.math.reduce_max(probs_temp) –  Nov 06 '19 at 06:03
  • You can use tf.Session() as sess, then print(prob_max, feed_dict={probs_temp: probs_temp}), be careful to check there is no other variables named prob_max, it may print the last variables named prob_max instead –  Nov 06 '19 at 06:05
  • My demo: https://colab.research.google.com/drive/13fZE0U3d7csINF27ksCTlBAKICQys6Z6, hope this helps :) –  Nov 06 '19 at 06:45
1

The input to K.log() gets too small. So, prevent prob_max from becoming so small:

prob_max = K.clip(prob_max, 1e-36, 1e36)
val = K.log(prob_max)

My experiment: 1e-36 is fine, but 1e-40 is too small.

Adrian Mole
  • 49,934
  • 160
  • 51
  • 83
Amir P
  • 123
  • 5