Why is the output different with Python 2.7 than Python 3.10?

Question

$ /usr/bin/python2 simple.py 200 > out2.pbm
$ /opt/src/Python-3.10.1/bin/python3 simple.py 200 > out3.pbm
$ cmp out2.pbm out3.pbm
out2.pbm out3.pbm differ: byte 304, line 3

The python2 output is correct. The python3 output is incorrect.

Here is a correct .pbm output file.

simple.py is

import sys

w = h = x = y = bit_num = 0
byte_acc = 0
i = 0; iterations = 50
limit = 2.0
Zr = Zi = Cr = Ci = Tr = Ti = 0.0

w = int(sys.argv[1])
h = w

sys.stdout.write("P4\n%d %d\n" % (w, h))

for y in range(h):

    for x in range(w):

        Zr = Zi = 0.0 
        Cr = (2.0 * x / w - 1.5); Ci = (2.0 * y / h - 1.0)        
        
        for i in range(iterations):

            Tr = Zr*Zr - Zi*Zi + Cr
            Ti = 2*Zr*Zi + Ci          
            Zr = Tr; Zi = Ti               
            if Zr*Zr+Zi*Zi > limit*limit:
                break
                        
        if Zr*Zr+Zi*Zi > limit*limit: 
            byte_acc = (byte_acc << 1) | 0x00
        else:
            byte_acc = (byte_acc << 1) | 0x01
                
        bit_num += 1         

        if bit_num == 8:
        
            sys.stdout.write(chr(byte_acc))            
            byte_acc = 0
            bit_num = 0

        elif x == w - 1:

            byte_acc = byte_acc << (8-w%8)
            sys.stdout.write(chr(byte_acc))  
            byte_acc = 0
            bit_num = 0

What changed that could cause the different output?

I re-opened this - somebody closed it pointing to a post about integer division, but best I can see there is no integer division done by this program. — Tim Peters, Mar 24 '22 at 20:41
Does https://stackoverflow.com/questions/48958212/equivalent-of-python2-chrint-in-python3 or https://stackoverflow.com/questions/30750830/how-can-i-duplicate-python-2-chr-exactly-in-python-3 answer your question? In short: `chr` in Python 2 and 3 doesn't output the same. But I wonder why you would print these characters. Are you trying to write bytes to some file? — Thierry Lathuille, Mar 24 '22 at 20:44
The issue is `chr`. To do the equivalent in Python 3, do `sys.stdout.buffer.write(bytes([byte_acc]))`. — Tim Roberts, Mar 24 '22 at 21:08
`w = h = x = y = bit_num = 0` -- reconsider that, in Python you don't declare variables like in C. Why does your code insist on using multiple expressions per line separated by a semicolon? You can use `a, b = 1, 2` instead. That said, pipe the output into `hd` or `hexdump` and compare the results. — Ulrich Eckhardt, Mar 24 '22 at 21:13
What is byte 304, line 3? Can you provide a [mre] that *just* outputs the difference? — MisterMiyagi, Mar 24 '22 at 21:13
Here's a minimal reproducible example under Windows: `python -c "print(chr(128))" > o1` — Tim Peters, Mar 24 '22 at 21:34
@Ulrich Eckhardt — it looks like C because the intention is to look line-by-line like a previous C program. — igouy, Mar 25 '22 at 05:29
@https://stackoverflow.com/users/1883316/tim-roberts sys.stdout.buffer.write(bytes([byte_acc])) and sys.stdout.flush() — igouy, Mar 25 '22 at 17:46

Tim Peters · Accepted Answer · 2022-03-25T16:50:07.690

1

I can't run this under Python 3.10.1 (Windows, 64-bit):

**Traceback (most recent call last):
  File ... simple.py", line 39, in <module>
    sys.stdout.write(chr(byte_acc))
  File  ... \lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\x80' in position 0: character maps to <undefined>

If I change the 2 instances of chr() to str() (so that it prints out a string representation of the byte's decimal value instead), it produces the same output under 3.10.1 and 2.7.11.

So you're getting burned by whatever default Unicode encoding scheme is used by your Python for sys.stdout under Python 3.

If I set an envar like so (syntax may differ under your OS):

set PYTHONIOENCODING=latin-1

then both Pythons produce the same output using chr().

One way

Here's one way to "fix it":

import sys
from sys import stdout

if hasattr(stdout, "buffer"): # Python >= 3
    def putbyte(b):
        assert 0 <= b < 256
        stdout.buffer.write(bytes([b]))
else: # before Python 3
    def putbyte(b):
        assert 0 <= b < 256
        stdout.write(chr(b))

and then change your code to use putbyte(byte_acc) instead of the current sys.stdout.write(chr(byte_acc)).

That's not quite enough, though. Writing to the internal buffer yourself also makes you responsible for buffer management across uses. After the current

sys.stdout.write("P4\n%d %d\n" % (w, h))

you also need to add

sys.stdout.flush()

to get the output string into the buffer before you add additional output bytes.

edited Mar 25 '22 at 16:50

answered Mar 24 '22 at 20:56

Tim Peters

67,464
13
126
132

>> If I change the 2 instances of chr() to str() (so that it prints out a string representation of the byte's decimal value instead), it produces the same output under 3.10.1 and 2.7.11.<< I see the same behaviour. Unfortunately, in that case, both produce the same incorrect output. Previously the Python 2.7 was correct and the Python 3.10 output was incorrect. – igouy Mar 25 '22 at 04:57
Write binary data to a file opened in binary mode,. You can follow various links to find ways to trick stdout into "acting like" it's in binary mode in Python 3, but life can be easy instead by not fighting it. – Tim Peters Mar 25 '22 at 05:21
The program output is required to be on stdout, not a file. – igouy Mar 25 '22 at 05:25
Then you're going to have to fight it. Follow links already given. The differences between text and binary mode cannot be ignored in a Unicode world (which Python 3 - but not Python 2 - lives in). – Tim Peters Mar 25 '22 at 05:32
1

The accepted answer should be considered: https://stackoverflow.com/questions/908331/how-to-write-binary-data-to-stdout-in-python-3 – Tim Peters Mar 25 '22 at 05:41
See edit at the end for workaround code. – Tim Peters Mar 25 '22 at 16:50
As you were commenting, I've been working through the previously suggested alternatives, and @tim-roberts suggestion seems to work with least fuss. – igouy Mar 25 '22 at 17:42
Yes, that's the same way as in the code I added. But it doesn't work under Python 2 (`sys.stdout` doesn't have a `buffer` attribute there). So the code I added detects which version is needed at runtime. – Tim Peters Mar 25 '22 at 17:45

Why is the output different with Python 2.7 than Python 3.10?

1 Answers1

One way