4

I'm currently porting a Python2 script to Python3 and have problems with this line:

print('\xfe')

When I run it with Python2 python test.py > test.out, than the file consists of the hex-values FE 0A, like expected.

But when I run it with Python3 python3 test.py > test.out, the file consists of the hex-values C3 BE 0A.

What's going wrong here? How can I receive the desired output FE 0A with Python3.

Jakube
  • 3,353
  • 3
  • 23
  • 40
  • FYI, that latter notation is the same textual character value, but in UTF8. That may give you a hint. – Jongware Aug 14 '15 at 19:30

3 Answers3

4

The byte-sequence C3 BE is the UTF-8 encoded representation of the character U+00FE.

Python 2 handles strings as a sequence of bytes rather than characters. So '\xfe' is a str object containing one byte.

In Python 3, strings are sequences of (Unicode) characters. So the code '\xfe' is a string containing one character. When you print the string, it must be encoded to bytes. Since your environment chose a default encoding of UTF-8, it was encoded accordingly.

How to solve this depends on your data. Is it bytes or characters? If bytes, then change the code to tell the interpreter: print(b'\xfe'). If it is characters, but you wanted a different encoding then encode the string accordingly: print( '\xfe'.encode('latin1') ).

dsh
  • 12,037
  • 3
  • 33
  • 51
  • I should probably take a good look at Unicode soon. But your two solutions don't work for me. I want to see (when calling `python3 test.py >test.out`) the hexvalue `FE` in test.out. – Jakube Aug 14 '15 at 19:43
  • No, the newline is fine. But I want the file to only contain two bytes (`FE 0A`) and not three bytes (`CE BE 0A`). – Jakube Aug 14 '15 at 19:49
  • Actually, you might not be able to do it with `print()`. `print()` is applying `repr()` to the object, since it isn't a string, then encoding the resultant string and writing that. – dsh Aug 14 '15 at 19:54
  • You really will want to understand characters and bytes very soon: that is one of the major fundamental differences in Python 3. Python 2 handled strings very sloppily, often confusing people as to whether they had bytes or characters they were handling. All `str` objects in Python 3 contain Unicode characters, and `bytes` is a separate data type. – dsh Aug 14 '15 at 19:56
  • @user38034: (1) to print text, always use Unicode in Python. Do not hard code the character encoding used by the current environment in your script (2) to print binary data such as image data, compressed data (gzip), encrypted data, see [How to write bytes to a file in Python 3 without knowing the encoding?](http://stackoverflow.com/q/4290716/4279) – jfs Aug 14 '15 at 23:58
4

print '\xfe' Python 2 code is roughly equivalent to this Python 3 code:

sys.stdout.buffer.write(b'\xfe' + os.linesep.encode())

while print('\xfe') Python 3 code is roughly equivalent to this Python 3 code:

sys.stdout.buffer.write((u'\xfe' + os.linesep).encode(sys.stdout.encoding))

In the first case Python prints bytes. In the second case, it prints Unicode and the result depends on your environment (locale).

>>> u'\xfe'.encode('utf-8')
b'\xc3\xbe'

To print text, always use Unicode in Python. Do not hardcode the character encoding used by the current environment in your script.

To print binary data such as image data, compressed data (gzip), encrypted data, see How to write bytes to a file in Python 3 without knowing the encoding?

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
1

The print(argument) converts the argument using str() (if neccessary) and then it calls file.write(string). The file is the optional argument of the print(), and it defaults to sys.stdout. It means that you should be able to do the same with sys.stdout.write(str(argument) + '\n'). So, the result depends on the used encoding that you can get from sys.stdout.encoding. If you pass another file argument, then the file object have to be open for writing in the text mode, and possibly a different encoding may be applied.

pepr
  • 20,112
  • 15
  • 76
  • 139