Getting an overflow error in stable sigmoid

Question

I have implemented the stable version of the sigmoid and yet I am getting overflow errors.

import numpy as np

f = lambda preds: (np.where(preds<0.,np.exp(preds)/(1+np.exp(preds)),1. / (1. + np.exp(-preds))))

arr = np.array([9999999999999999999999999999.,-99999999999999999999999999999999999.])
f(arr)
output:
<ipython-input-3-2b9a4fbf3d68>:3: RuntimeWarning: overflow encountered in exp
  f = lambda preds: (np.where(preds<0.,np.exp(preds)/(1+np.exp(preds)),1. / (1. + np.exp(-preds))))
<ipython-input-3-2b9a4fbf3d68>:3: RuntimeWarning: invalid value encountered in true_divide
  f = lambda preds: (np.where(preds<0.,np.exp(preds)/(1+np.exp(preds)),1. / (1. + np.exp(-preds))))
array([1., 0.])

What am I misunderstanding here? Is it that NumPy evaluates both the branches like tensorflow.cond in tensorflow 1 used to?

No such warnings here:

arr = np.array([9999999999999999999999999999.,-99999999999999999999999999999999999.])
for preds in arr:
    if preds < 0.:
        np.exp(preds)/(1+np.exp(preds))
    else:
        1. / (1. + np.exp(-preds))

Of course it evaluates both branches but if the wrong value is discarded then you don't need to worry... — user202729, Sep 03 '21 at 10:37
"Is it that NumPy evaluates both the branches" - yes. `np.where` is a regular function, so all of its arguments must be evaluated before calling it — ForceBru, Sep 03 '21 at 10:37
This question [python - Numpy "Where" function can not avoid evaluate Sqrt(negative) - Stack Overflow](https://stackoverflow.com/questions/52622172/numpy-where-function-can-not-avoid-evaluate-sqrtnegative) is more general but contains the solution to the general issue (use a mask). Although for this particular case (sigmoid) there's a better solution, see above — user202729, Sep 03 '21 at 10:39
FYI: Instead of creating your own logistic sigmoid function, you can use the [`expit`](https://scipy.github.io/devdocs/reference/generated/scipy.special.expit.html) function from SciPy. — Warren Weckesser, Sep 03 '21 at 12:25
*"Is it that NumPy evaluates both the branches..."* Actually it is *Python* that evaluates both arguments, just like it would with any other function. — Warren Weckesser, Sep 03 '21 at 12:31

Getting an overflow error in stable sigmoid

0 Answers0