Problem with precision in a python script

Question

I need to convert numbers to a flat file format 0.57 becomes 000000000000000057 in this format (padding zeroes) My method

My method

def toAmount18(a):
   return str(int(a*100)).zfill(18)

For 0.57 it outputes 000000000000000056

The problem lies in a*100 it outputs 56.99999999999999 instead of 57. Ideas?

score 1 · Accepted Answer · answered Dec 18 '20 at 10:44

1

It is a floating point accuracy problem. In the early days of Fortran IV, we learned that the conversion from a float to an integer value was I = F + .5

Here you can still use the same trick:

def toAmount18(a):
   return str(int(a*100 + .5)).zfill(18)

or use more modern tools like round

   return str(round(a*100)).zfill(18)

The nice point with the manual method is that you can specify the truncation magnitude to your needs:

  return str(int(a*100 + .00005)).zfill(18)

will only round after the 4th decimal.

answered Dec 18 '20 at 10:44

Serge Ballesta

143,923
11
122
252

I tried to cast it to a decimal first which didn't help. We do not have this problem in c#. It seems pythons floating point precision isn't up to date :) – Anders Dec 18 '20 at 11:01
IEEE floating point is the same for all languages. But maybe C# has added some rounding magic at the last bits of the mantissa. As far as I am concerned, I prefere to decide myself which rounding I want to use than rely on a language feature, unless it is clearly documented (which is possible, I do not use C#... ) – Serge Ballesta Dec 18 '20 at 11:10
C# complies with IEEE https://learn.microsoft.com/en-us/cpp/build/ieee-floating-point-representation?view=msvc-160 interesting that 0.57 x 100 turns out 57 there. Need to investigate. – Anders Dec 18 '20 at 11:24
@Anders: The IEEE says nothing about how you convert a floating value to an int. But whatever the language `(0.57 * 100) < 57.0` will be true. – Serge Ballesta Dec 18 '20 at 11:26
This is interesting in.NET5 0.57x100 is 56.9999 but in net core 3.1 its 57. This needs digging, they must have removed a automatic rounding in net 5. Oh well its out of scope for question. Thanks – Anders Dec 18 '20 at 11:34

CastelPablito · Answer 2 · 2020-12-18T10:38:16.197

0

Using int() as default works like a floor on the input number.

Try to use round() method instead of int() as shown in example: https://www.programiz.com/python-programming/methods/built-in/round

You can also add an extra security layer (try statement) which will handle wrong type exceptions.

edited Dec 18 '20 at 10:38

answered Dec 18 '20 at 10:35

CastelPablito

1
2

round can lead to unwanted rounding though. We want truncate behavior of int 0,579 should still be treated as 57 on other end – Anders Dec 18 '20 at 10:38
like what? :D You missed something? – Anders Dec 18 '20 at 10:43
in this case, you can do something like this: 1) b = a * 100 - 57,9 2) c = floor(b) - now you got 57 3) d = b - c - here you got 0,9 4) e = b - d - now you got 57 again You can simplify this operation and optimalise it with an if statement – CastelPablito Dec 18 '20 at 10:46

score 0 · Answer 3 · answered Dec 18 '20 at 11:15

You could avoid maths altogether and invite regex to the party:

re.search(r'\.(\d{,2})', str(0.5)).group(1).zfill(18)
'000000000000000005'

re.search(r'\.(\d{,2})', str(0.57)).group(1).zfill(18)
'000000000000000057'

re.search(r'\.(\d{,2})', str(0.571)).group(1).zfill(18)
'000000000000000057'

re.search(r'\.(\d{,2})', str(0.579)).group(1).zfill(18)
'000000000000000057'

Might need some more sample inputs to get this tuned correctly for the expected output.

Problem with precision in a python script

3 Answers3