logistic / sigmoid function implementation numerical precision

Question

in scipy.special.expit, logistic function is implemented like the following:

if x < 0
    a = exp(x) 
    a / (1 + a) 
else 
    1 / (1 + exp(-x))

However, I have seen implementations in other languages/frameworks that simply do

1 / (1 + exp(-x))

I am wondering how much benefit the scipy version actually brings.

For very small x, the result approaches to 0. It works even if exp(-x) overflows to Inf.

It seems that none of the answers given actually address the question. Is `1 / (1 + exp(-x))` accurate or not? — a06e, May 03 '20 at 00:10

score 10 · Accepted Answer · edited May 23 '17 at 12:01

10

It's really just for stability - putting in values that are very large in magnitude might return unexpected results otherwise.

If expit was implemented just as 1 / (1 + exp(-x)) then putting a value of -710 into the function would return nan, whereas -709 would give a value close to zero as expected. This is because exp(710) is too big to be a double.

The branching in the code just means that this scenario is avoided.

See also this question and answer on Stack Overflow.

edited May 23 '17 at 12:01

Community

1
1

answered May 06 '16 at 14:50

Alex Riley

169,130
45
262
238

I'm not sure I follow your point... stability is cited as the motivation for the code in [this GitHub thread](https://github.com/scipy/scipy/pull/3386). – Alex Riley May 06 '16 at 15:21
5

What I was trying to say, `1 / (1 + exp(-710))` doesn't return `NaN`, it returns 0 in IEEE standard as `1 / Inf = 0`. And it is true in numpy `1 / (1 + np.exp(-710)) == 0` – colinfang May 06 '16 at 16:24
1

But the problem is surely for `expit(x)` where `x` is some large-magnitude *negative* value. For instance `expit(-710)` needs to calculate the exponent of positive 710, i,e. `exp(+710)`. The `exp` in the source code I *think* refers to your system's C math library `exp` (not the ufunc `np.exp`) which will raise an error if overflow is detected, as is the case here. – Alex Riley May 06 '16 at 16:51

score 4 · Answer 2 · answered Jul 20 '16 at 09:34

4

Seems it would be more efficient to use:

if x < -709
  sigmoid = 0.0
else
  sigmoid = 1.0 / (1.0 + exp(-x))

unless you need number with 10^-309 precision (see below) which seems overkill!

>>> 1 / (1 + math.exp(709.78))
5.5777796105262746e-309

answered Jul 20 '16 at 09:34

Eamonn Kenny

1,926
18
20

Not necessary, since `exp(710)` evaluates to infinity and `1 / (1 + inf) = 0` in floating point – a06e May 03 '20 at 00:10

score 1 · Answer 3 · answered Feb 01 '17 at 18:15

Another way to do it would be

python np.where(x > 0, 1. / (1. + np.exp(-x)), np.exp(x) / (np.exp(x) + np.exp(0)))

Since np.exp(x) / (np.exp(x) + np.exp(0)) is equivalent to 1. / (1. + np.exp(-x)) but more stable for negative values

logistic / sigmoid function implementation numerical precision

3 Answers3

Linked