3

I have a binary classification problem. I am using the log_loss from tensorflow.losses.log_loss.

To check, I use sklearn.metrics.log_loss. Most of the times, the two functions give the same result (only difference in dtype). In some instance, the sklearn function returns NaN while tf.losses.log_loss returns a correct value.

data is here: https://pastebin.com/BvDgDnVT

code:

import sklearn.metrics
import tensorflow as tf
y_true = [... see pastebin link]
y_pred = [... see pastebin link]
loss_sk = sklearn.metrics.log_loss(y_true, y_pred, labels=[0, 1]) # -> returns NaN
with tf.Session() as sess:
    loss_tf = tf.losses.log_loss(y_true, y_pred).eval(session=sess) # -> returns 0.0549

There seems to be some log(0) happening, but why does tensorflow not have this problem?

skjerns
  • 1,905
  • 1
  • 16
  • 25
  • This looks like you've stumbled across a bug as I get the same result. It looks like some form of overflow, as taking subsections of the list works fine, so it's not one particular value. – piman314 May 03 '18 at 16:09

2 Answers2

6

Changing the dtype of both arrays to a 64-bit float fixes it

dtype=np.float64

for example adding y_pred = y_pred.astype(np.float64)

Roelant
  • 4,508
  • 1
  • 32
  • 62
Jan K
  • 4,040
  • 1
  • 15
  • 16
  • Why? I just stumbled on this problem too, but curiously converting my numpy arrays to pandas Series of dtype float64 fixed it! – Corey Levinson Apr 12 '19 at 17:07
3

Another way of fixing the issue is to provide eps=1e-7 to log_loss, which is a more appropriate epsilon for float32 and is what tensorflow's using. Scikit however uses 1e-15 as a default (expecting float64).

Disenchanted
  • 565
  • 6
  • 16
  • Thank you! I was using scikit-learn `permutation_importance` and gets NaNs for importance because of this. Setting eps in scorer fixes the issue. :) – pplonski Jul 09 '20 at 09:17