10

I'm trying to calculate the entropy of the weights while training my graph and to use it for regularization. This of course involves w*tf.log(w), and as my weights are changing some of them are bound to get into a region which results in NaNs being returned.

Ideally I would include a line in my graph setup:

w[tf.is_nan(w)] = <number>

but tensorflow doesn't support assigning like that. I could of course create an operation, but that wouldn't work because I need for it to happen during the execution of the entire graph. I can't wait for the graph to execute and then 'fix' my weights, is has to be part of the graph execution.

I haven't been able to find an equivalent to np.nan_to_num in the docs.

Anybody have an idea?

(For obvious reasons, adding an epsilon doesn't work)

Nimitz14
  • 2,138
  • 5
  • 23
  • 39
  • [Try to avoid NaNs in the first place](http://stackoverflow.com/questions/33712178/tensorflow-nan-bug/42497444#42497444). – jvdillon Jul 21 '17 at 19:44

3 Answers3

26

I think you need to use tf.select.

w = tf.select(tf.is_nan(w), tf.ones_like(w) * NUMBER, w); #if w is nan use 1 * NUMBER else use element in w

Update: TensorFlow 1.0 has deprecated tf.select in favor of Numpy compatible tf.where.

musically_ut
  • 34,028
  • 8
  • 94
  • 106
chasep255
  • 11,745
  • 8
  • 58
  • 115
  • 6
    w = tf.where(tf.is_nan(w), tf.ones_like(w) * NUMBER, w) # version with tf.where - to copy faster and avoid confusion – Andy R Jul 02 '18 at 09:36
  • 3
    For Tensorflow 2.0+ you need to use tf.math.is_nan. This code removes nans that where created by the padding portion of categorically_cross_entropy loss function: `loss_value = tf.where(tf.math.is_nan(loss_value), tf.zeros_like(loss_value), loss_value)` – Behdad Forghani Mar 13 '20 at 11:36
0

I use this to convert a value to 0 if it's NaN:

value_not_nan = tf.dtypes.cast(tf.math.logical_not(tf.math.is_nan(value)), dtype=tf.float32)
tf.math.multiply_no_nan(value, value_not_nan)
John Targaryen
  • 1,109
  • 1
  • 13
  • 29
-4

You cannot convert nan to a number (such as you cannot convert infinite to a number).

The nan resulkts most probably from the w*tf.log(w) once w is (or contains) zero(s). You might add 1e-6 first so that no division by zero occurs.

Phillip Bock
  • 1,879
  • 14
  • 23