Python arithmetic with small numbers

Question

I am getting the following unexpected result when I do arithmetic with small numbers in Python:

>>> sys.float_info
sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1)
>>> (1. - (1.e-17) ) < 1.
False

I know that floating point numbers do not have infinite precision, but it should be able to handle "large" small numbers like 1e-17, shouldn't it?

Answer 1

The precision of floats is higher near 0 than it is near 1.

• There are 4591870180066957722 floats between 0.0 and 0.1.
• There are 900719925474099 floats between 0.9 and 1.0, fewer by far.

Float density halves at regular intervals, it looks something like this:

Here is the next float after 1., in the direction of 0.:

>>> import math
>>> math.nextafter(1., 0.)
0.9999999999999999
>>> format(math.nextafter(1., 0.), ".32f")  # let's see more decimal places
'0.99999999999999988897769753748435'

The mathematically correct value of 1 - 10^-17 is 0.99999999999999999 (there are seventeen nines), I'll call this number n. Like almost all numbers, n can't be represented exactly with a float.

0.99999999999999999                    # n
0.00000000000000001                    # distance between n and 1, i.e. 10^-17
0.00000000000000010102230246251565...  # distance between n and nextafter(1., 0.)

So you see, 1 - 10^-17 is about 10 times further from nextafter(1., 0.) than it is from 1.. When the expression 1. - 1.e-17 is evaluated by the interpreter it gives you back the closest possible result, which is 1. exactly. It wouldn't make sense to return any other float, that would be even further away from the "real" result (pardon the pun).

Note: math.nextafter is available in Python 3.9+. In earlier versions you can use numpy.nextafter similarly.

Related question -> Increment a Python floating point value by the smallest possible amount

Answer 2

First, let's review what epsilon really is in the return value of sys.float_info.

Epsilon (or ) is the smallest number such that 0.5 + ≠ 0.5 AND 0.5 - ≠ 0.5

Python is telling you that the smallest number that will cause 0.5 to increment or decrement repeatably is epsilon=2.220446049250313e-16 -- but this is only for the value 0.5. You are attempting to increment 1.0 by 1.0e-17. This is a larger value (1.0 vs 0.5) being incremented by a smaller number than the for 0.5 (1.0e-17 vs 2.2e-16). You are off by an order of magnitude roughly, since the increment value of 1.0e-17 is an order of magnitude smaller than the relative epsilon for 1.0.

You can see this here:

These change the value of 0.5

>>> 0.5+sys.float_info.epsilon
0.5000000000000002
>>> 0.5-sys.float_info.epsilon
0.4999999999999998

These values do not:

>>> 0.5+sys.float_info.epsilon/10.0
0.5
>>> 0.5-sys.float_info.epsilon/10.0
0.5
>>> 5.0+sys.float_info.epsilon
5.0
>>> 5.0-sys.float_info.epsilon
5.0

Explanation:

IEEE 754 defines the floating point format in use today on most standard computers (specialty computers or libraries may use a different format.) The 64 bit format of IEEE 754 uses 53 bits of precision to calculate and 52 to store to the mantissa of a floating point value. Since you have a fixed 52/53 bits to work with, the magnitude and accuracy of the mantissa changes for larger / smaller values. So then the changes as the relative magnitude of a floating point number changes. The value of for 0.5 is different that the value for 1.0 and for 100.0.

For a variety of very good and platform-specific reasons (storage and representation, rounding, etc), even though you could use a smaller number, epsilon is defined as using 52 bits of precision for the 64 bit float format. Since most Python implementations use a C double float for float, this can be demonstrated:

>>> 2**-52==sys.float_info.epsilon
True

See how many bits your platform will do:

>>> 0.5 + 2.0**-53
0.5000000000000001
>>> 0.5 - 2.0**-53
0.4999999999999999
>>> 0.5 + 2.0**-54
0.5                           # fail for 0.5 + 54 bits...
>>> 0.5 - 2.0**-54
0.49999999999999994           # OK for minus
>>> 0.5 - 2.0**-55
0.5                           # fail  for 0.5 minus 55 bits...

There are several work arounds for your issue:

1. You can use the C99 concept of nextafter to calculate the value appropriate epsilon. For Python, either use numpy or the Decimal class to calculate nextafter. More on nextafter in my previous answer HERE
2. Use integers. A 64 bit integer will clearly handle an epsilon value in the 17th order of magnitude without rounding.
3. Use an arbitrary precision math library. Decimal is in the standard Python distribution.

The important concept is that the value of is relative to value (and if you are incrementing or decrementing).

This can be seen here:

>>> numpy.nextafter(0.0,1.0)-0.0
4.9406564584124654e-324                 # a relative epsilon value of 4.94e-324
>>> numpy.nextafter(0.01,1.0)-0.01
1.7347234759768071e-18                  # 1e-17 would still work...
>>> numpy.nextafter(0.1,1.0)-0.1
1.3877787807814457e-17                  # 1e-17 would >>barely<< work...
>>> numpy.nextafter(0.5,1.0)-0.5
1.1102230246251565e-16                  # a relative epsilon value of 1.1e-16
>>> numpy.nextafter(500.0,501.0)-500.0
5.6843418860808015e-14                  # relative epsilon of 5.6e-14
>>> numpy.nextafter(1e17,1e18)-1e17
16.0                                    # the other end of the spectrum...

So you can see that 1e-17 will work handily to increment values between 0.0 and 0.1 but not many values greater than that. As you can see above, the relative for 1e17 is 16.

Answer 3

it should be able to handle "large" small numbers like 1e-17, shouldn't it?

Not necessarily (it depends on the numbers). A float cannot exactly represent either 1e-17 or 1-(1e-17). In the case of the latter, the nearest number that it can represent is 1.

I suggest you read What Every Computer Scientist Should Know About Floating-Point Arithmetic.

Answer 4

If you need this level of precision, consider the Decimal module

>>> decimal.Decimal(1.0)-decimal.Decimal('1.0e-17')
Decimal('0.999999999999999990')
>>> decimal.Decimal(1.0)-decimal.Decimal('1.0e-17')<decimal.Decimal(1.0)
True

And:

>>> decimal.Decimal(1.0)-decimal.Decimal('1.0e-17')<1.0
True

Careful with the last one though because you can get conversion errors.

Others have suggested What Every Computer Scientist Should Know About Floating-Point Arithmetic. and I also recommend Don’t Store That in a Float

Answer 5

you can handle those. note that

>>> 1.e-17 == 0
False

and

>>> 1.e-17 + 1.e-18
1.1e-17

you simply cannot handle 1-1e-17, because the mantissa won't fit in the finite precision

Answer 6

>>> from decimal import Decimal
>>> Decimal('1')-Decimal(str(10**-17)) < Decimal('1')
True

Use the decimal module, for that kind of accuracy!

Answer 7

You have epsilon=2.220446049250313e-16 so it is normal that (1. - (1.e-17) ) = 1 because 1.e-17 < epsilon.