Sigmoid function returns 1 for large positive inputs

Question

I wrote the following function in Python to calculate sigmoid function of a scalar, vector or matrix.

def sigmoid(z):
    sig = 1.0/(1.0 + np.exp(-z))
    return sig

For relatively large positive values of z, e^-z returns a very small value close to zero (0) and hence the value of sig is rounded to 1. My final objective is to determine cost function of logistic regression algorithm. Since sigmoid returns exactly 1, log(1-1) return 'nan'. How can I solve the problem in such a way that my sigmoid() function will return the proper value and not round the e^-z to 0?

When I normalized the input features using mean and standard deviation, it worked fine. But is there a way to make it work with larger values of z?

Also, I tried the same on Matlab and it worked fine without normalization.

>>> Z = np.array([[60, 100],[20, 80]])
>>> Z
array([[ 60, 100],
       [ 20,  80]])
>>> np.exp(-Z)
array([[8.75651076e-27, 3.72007598e-44],
       [2.06115362e-09, 1.80485139e-35]])
>>> 1.0/(1.0 + np.exp(-Z))
array([[1., 1.],
       [1., 1.]])

It is never rounded, can you show an example? Specifically, what is the input value of `z` that makes the fucntion _round_? — Evgeny, Sep 20 '18 at 10:45
It is not "wrong" that the function returns 1, it is just that you have reached the machine precision limit. Your cost function should be "protected" to this, either having a minimum value for the sigmoid (`log(1 - max(sigmoid(z), EPSILON))`) or defining it on a conditional basis, such that if `sigmoid(z) == 1` (or `sigmoid(z) > 1 - EPSILON`) then the cost is zero. — jdehesa, Sep 20 '18 at 10:49
@EPo: Hi, added a snippet from console. I'm not sure if I'm doing or interpreting something wrong. — Supratim Haldar, Sep 20 '18 at 11:04
@jdehesa: Hi, thanks for the explanation, this make sense! I'll try this out. — Supratim Haldar, Sep 20 '18 at 11:10
Check out this [question](https://stackoverflow.com/questions/37074566/logistic-sigmoid-function-implementation-numerical-precision) on how to implement sigmoid robustly. — lightalchemist, Sep 20 '18 at 11:40
Thanks @lightalchemist. But I don't think that approach can solve this particular problem. — Supratim Haldar, Sep 21 '18 at 09:23
I could solve the problem by following @jdehesa's suggestion. Added the following before returning sig: sig[sig == 1.0] = 0.9999 and sig[sig == 0.0] = 0.0001 — Supratim Haldar, Sep 21 '18 at 09:24

score 4 · Accepted Answer · answered Mar 26 '19 at 18:55

I overcame this issue by wrapping the sigmoid function with np.minimum & np.maximum:

def sigmoid(x):
    sig = 1 / (1 + np.exp(-x))     # Define sigmoid function
    sig = np.minimum(sig, 0.9999)  # Set upper bound
    sig = np.maximum(sig, 0.0001)  # Set lower bound
    return sig

As a result my losses started looking like this

But both training and test losses converged well & I received ~90% of accuracy on a tiny dataset using logistic regression.

Andrew McDowell · Answer 2 · 2018-09-20T11:42:12.453

As already mentioned by jdehesa, your issue is with precision limits. You can read more here: https://docs.python.org/2/tutorial/floatingpoint.html

You could try using the Decimal class which avoids rounding:

from decimal import Decimal
import numpy as np
import math

def sigmoid(z):
    sig = Decimal(1.0)/(Decimal(1.0) + Decimal(np.exp(-z)))
    return sig


math.log(Decimal(1)-sigmoid(60))
>>> -59.97257293350302

but this will only work up to a point (I see it already fails for 80). I did some reading and if you really need more precision, you can change the precision of the Decimal object by increasing the value of the following from the default of 28:

from decimal import *
getcontext().prec = 28

More detail here:

https://docs.python.org/2/library/decimal.html

For most cases though, the suggestion of manually handling small results with a logic check will probably be best.

Hi Andrew, thanks for the suggestions, these were helpful! However, I could solve the problem by following @jdehesa's suggestion. Added the following before returning sig: sig[sig == 1.0] = 0.9999 and sig[sig == 0.0] = 0.0001. — Supratim Haldar, Sep 21 '18 at 09:29

Sigmoid function returns 1 for large positive inputs

2 Answers2