Why can "%.10f" % Decimal(u) emit a string with a literal colon?

Question

When formatting a number to be printed, 12 digit numbers are being formatted with a colon immediately after the dot. Why is this happening? This is Python 2.7 on an AIX system.

$ uname -a ; /opt/bin/python2.7
AIX myserver 1 6 00F6A5CC4C00
Python 2.7.12 (default, Sep 29 2016, 12:02:17) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> '{0:.10f}'.format(123456789012)
'123456789011.:000000000'
>>> from decimal import Decimal
>>> u=123456789012
>>> print "%.10f" % Decimal(u)
123456789011.:000000000

Further information:

It is not every 12 digit number:

>>> for x in range(123456789010,123456789020):
...     print '{0:.10f}'.format(x)
...
12345678900:.0000000000
123456789010.:000000000
123456789011.:000000000
123456789013.0000000000
123456789013.:000000000
123456789015.0000000000
123456789016.0000000000
123456789017.0000000000
123456789017.:000000000
123456789019.0000000000

This is not happening with any other length numbers. Also, I tried bash's and perl's printf and this is not happening with either of them.

What is happening here?

As requested, here is a screen shot video.

More requested information:

>>> import locale
>>> locale.getdefaultlocale()
('en_US', 'ISO8859-1')

Result of user2357112 pastebin code:

>>> import ctypes
>>> f=ctypes.pythonapi.PyOS_double_to_string
>>> f.argtypes=ctypes.c_double,ctypes.c_char,ctypes.c_int,ctypes.c_int,ctypes.POINTER(ctypes.c_int))
>>> f.restype=ctypes.c_char_p
>>> print f(123456789012.0, 'f', 10, 0, None)
123456789011.:000000000

Antti_Happa's pastebin printed all numbers correctly.

Using format r gives:

print 'e: {0:.10e}\nf: {0:.10f}\ng: {0:.10g}\nr: {0:0r}'.format(x)
ValueError: Unknown format code 'r' for object of type 'int'

Using e, f and g formats provided the following:

for x in range(123456789000,123456789019):
print 'e: {0:.10e}\nf: {0:.10f}\ng: {0:.10g}'.format(x)
e: 1.2345678900e+11
f: 123456789000.0000000000
g: 1.23456789e+11
e: 1.2345678900e+11
f: 123456789000.:000000000
g: 1.23456789e+11
e: 1.2345678900e+11
f: 123456789001.:000000000
g: 1.23456789e+11
e: 1.2345678900e+11
f: 123456789003.0000000000
g: 1.23456789e+11

I have no access to install or update anything on this server. I can request an updated version, but change requests of this nature take a fair amount of time. Also, other programs depend on this installation and a lot of testing would be required.

I have been informed that only IBM provided packages will be installed and that the latest python 2.7 package provided by IBM is 2.7.12.

I have "fixed" the problem by doing

othervar = '{0:.10f}'.format(somevar).replace(':', '0')

which is extremely unsafe, I know, but ... shrug

Argh! I just noticed an off-by-one error ... 123456789012 is being formatted as one less: 123456789011.:0000000000 ... this is a weird bug.

We don't usually ask for this, but can you post a screenshot of this behavior? — user2357112, Aug 24 '17 at 21:58
2.7.12 isn't the latest python2 version ... can you try with 2.7.13 perhaps? And are you using vendor patches or upstream? — o11c, Aug 25 '17 at 17:13
@wim that doesn't make sense: Docker only does containers, but AIX is a different OS so it requires a VM. — o11c, Aug 25 '17 at 17:21
Warning: your "fix" is missing the fact that the string has more wrong with it than just a colon. Specifically, the last `2` is becoming a `1`. — user2357112, Aug 25 '17 at 17:32
For anyone else who wants to dig into the source, [here's](https://hg.python.org/cpython/file/v2.7.12) the 2.7.12 Python source tree. I suspect the problem may eventually be traced to a bug in the AIX implementation of some fundamental library routine Python relies on. — user2357112, Aug 25 '17 at 17:38
Could you run the contents of [this Pastebin](https://pastebin.com/CbBNEegx) and show us the results? It should help narrow down the exact function where the problem occurs. (It also leaks memory, but that doesn't matter for this test.) — user2357112, Aug 25 '17 at 17:52
Added the results, commenting here in case it's lost in the now long question. — harleypig, Aug 25 '17 at 18:04
All right, that narrows things down. I'm kind of suspecting something wrong in AIX's `vsnprintf`, but the fact that Bash and Perl aren't exhibiting the problem makes that less likely. — user2357112, Aug 25 '17 at 18:09
@harleypig can you try all of the codes `'e', 'f', 'g'`; and also the `r` with precision 0. — Antti Haapala -- Слава Україні, Aug 25 '17 at 19:10
@harleypig could you try [this one](https://pastebin.com/hg7udNqK) it gets one call deeper... (still prints 123456789012.0000000000) — Antti Haapala -- Слава Україні, Aug 25 '17 at 19:22
The `:` character occurs in ascii right after the `9`. In fact if you treated these `:`'s as numerically equivalent to a `10` in that spot, the output would be technically correct. — Aaron, Aug 28 '17 at 22:01
@AaronTaggart Yes looks like unintended overflow or rounding error inside python `print` implementation. I am not Python user but this sort of things usually happen due to rounding error (wrong rounding and its edge case) while fraction is multiplied by base during conversion to string. Another reason can be multi-threading issues. — Spektre, Aug 29 '17 at 16:03
This sounds like a float formatting bug that only manifests on AIX. Perhaps consider reporting it at https://bugs.python.org/issue? — Marius Gedminas, Aug 30 '17 at 10:28
Responded to Antti_Haapala's request for information, commenting here in case it gets lost above. Also, how do I find out if it has vsnprintf, 'man vsnprintf' didn't show anything. — harleypig, Aug 31 '17 at 22:29
...huh. PyOS_double_to_string showed the bug, but the _PyOS_ascii_formatd attempt didn't? Not what I expected. I wish I had access to a system I could reproduce this on, but AIX is proprietary, and exposing the actual system you're testing this on to random strangers on the internet is a Bad Idea. — user2357112, Sep 01 '17 at 00:08
you don't need a youtube for a screenshot. Next time just use http://snag.gy — pashute, Sep 11 '17 at 08:54
[See here for similar question](https://stackoverflow.com/questions/16612301/python-inserts-a-colon-in-a-decimal-number-from-access-via-pyodbc) [And here for another similar question](https://stackoverflow.com/questions/13825487/python-float-got-a-colon-after-the-decimal-point-after-addition) sadly, closed as "not a real question" — pashute, Sep 11 '17 at 09:14
What happens when you try with following option? `print "%.10f" % 123456789012`. This works even under `python 2.5` and should output: `123456789012.0000000000` — Cedric Zoppolo, Sep 15 '17 at 16:22
This happens regardless of how I print it out. As a matter of fact, that was how I originally printed it out and discovered this problem. — harleypig, Sep 15 '17 at 19:29
Dee Holtsclaw [reports](https://stackoverflow.com/a/46554601/) being unable to reproduce this on aix7 and Python 2.7.13. This seems to be the first input we've had from another AIX user. Pastebin reproduction of his terminal session [here](https://pastebin.com/MJWbD0Xh), because he can't comment and had to post it as an answer, and the answer will probably be deleted. — user2357112, Oct 03 '17 at 23:01
@OP: you should recompile your Python with debug info (without optimalization) and find the problem. (I don't have IBM-xlc; with my gcc-compiled Python-2.7.11 I couldn't reproduce the problem.) — Lorinczy Zsigmond, Oct 04 '17 at 05:44
Or at least you could quote the output of `python2-config --cflags` eg `-I/usr/local/include/python2.7 -I/usr/local/include/python2.7 -fno-strict-aliasing -mtune=native -maix64 -std=c99 -fPIC -DPIC -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes` — Zsigmond Lőrinczy, Oct 04 '17 at 13:37
Floating-point numbers being involved might be part of the problem; here is a blog entry I've just found: https://www.ibm.com/developerworks/community/blogs/5894415f-be62-4bc0-81c5-3956e82276f3/entry/XLC_bug_with_simple_floating_point_caculation_Notes_for_porting_floating_point_application_to_xlc?lang=en — Zsigmond Lőrinczy, Oct 04 '17 at 13:44
It seems to be specific to that version of python (2.7.12) an only on AIX. I was able to use this question and answer as a support for updating python and the problem went away with python 2.7.13. — harleypig, Jul 06 '18 at 00:17

score 1 · Answer 1 · answered Oct 03 '17 at 22:52

1

Although not an "answer", I can give you my results on a slightly newer version running on AIX.

Sorry that I'm unable to replicate your problem.

[lholtscl@ibm3 ~]$ python
Python 2.7.13 (default, Sep  7 2017, 21:08:50) [C] on aix7
Type "help", "copyright", "credits" or "license" for more information.
>>> print "%.10f" % 123456789012
123456789012.0000000000
>>> '{0:.10f}'.format(123456789012)
'123456789012.0000000000'

answered Oct 03 '17 at 22:52

Dee Holtsclaw

56
5

Thank you. I can actually use your comment to push for updating python. – harleypig Oct 03 '17 at 23:14

Why can "%.10f" % Decimal(u) emit a string with a literal colon?

1 Answers1

Linked