Why does Python 3 output \xe3, an extra char?

Question

Why does Python add \xe3 in the output of:

>>> b'Transa\xc3\xa7\xc3\xa3o'.decode('utf-8')
'Transaç\xe3o'

Expected value is:

'Transação'

Some more information about my environment:

>>> import sys
>>> print (sys.version)
3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)]   
>>> sys.stdout.encoding
'cp437'

This was under Console 2 + Powershell.

You need to tell your operating system, Python version and terminal you are using (grab version string from `python` startup) — Mikko Ohtamaa, Jun 08 '15 at 19:15
It has 2 problems when I test it. The output is: `u'Transa\xe7\xe3o'` — ROMANIA_engineer, Jun 08 '15 at 19:16
What specific version of Python is this? What characterset does your console support? — Martijn Pieters, Jun 08 '15 at 19:16
@helpYou: that's because you are testing with Python 2, not 3. Do pay attention! — Martijn Pieters, Jun 08 '15 at 19:16
It seems you are using ms-windows. If you are using `cmd.exe`, to need to change the code page (`chcp 65001`), and get cmd.exe to use a unicode capable font. See e.g [this question](http://stackoverflow.com/questions/388490/unicode-characters-in-windows-command-line-how/388500#388500). — Roland Smith, Jun 08 '15 at 19:27
@RolandSmith: `chcp 65001` raises more problems still though. — Martijn Pieters, Jun 08 '15 at 19:41
@MartijnPieters My solution for stuff that has to work in a terminal on windows is to accept that it sucks, force ASCII output on that platform and save myself the hassle of dealing with it. :-) — Roland Smith, Jun 08 '15 at 19:49
@RolandSmith: yup, I try to avoid the Windows console altogether (or, for that matter, Windows *full stop*). — Martijn Pieters, Jun 08 '15 at 19:50
Windows shell is broken that's not even funny. I hurt myself on an hourly basis... sigh. — Igor Gatis, Jun 09 '15 at 01:10

Martijn Pieters · Accepted Answer · 2015-06-09T18:06:00.223

5

You need to use a console or terminal that supports all of the characters that you want to print.

When printing in the interactive console, the characters are encoded to the correct codec for your console, with any character that is not supported using the backslashreplace error handler to keep the output readable rather than throw an exception. This is a feature of the default sys.displayhook() function:

If repr(value) is not encodable to sys.stdout.encoding with sys.stdout.errors error handler (which is probably 'strict'), encode it to sys.stdout.encoding with 'backslashreplace' error handler.

Your console can handle ç but not ã. There are several codecs that include the first character but not the last; you are using IBM codepage 437, but it is by no means the only one.

If you are running Python in the standard Windows console (cmd.exe) then be aware that Python, Unicode and that console do not mix very well. You can install the win-unicode-console package to make Python 3 use the Windows APIs to better output Unicode text; you'll need to make sure you have a font capable of displaying your Unicode text still.

I don't know for certain if that package is compatible with other Windows shells; your mileage may vary.

edited Jun 09 '15 at 18:06

answered Jun 08 '15 at 19:19

Martijn Pieters

1,048,767
296
4,058
3,343

From the look of version string, the OP is running on ms-windows, probably in `cmd.exe`. Not the best terminal out there. Out of the box, it will most definitely *not* be using UTF-8. How to change that has been answered several times on SO. – Roland Smith Jun 08 '15 at 19:34
Windows console allows you to print Unicode characters that are unsupported by the current code page. `win-unicode-console` that you've mentioned does it. See [python3 print unicode to windows xp console encode cp437](http://stackoverflow.com/q/28521944/4279) – jfs Jun 09 '15 at 17:59
you should probably mention that the "backslashreplace" behavior if true is specific to `repr()` (used by `sys.displayhook()` in REPL) – jfs Jun 09 '15 at 18:01
@J.F.Sebastian: Yes, this is `sys.displayhook()` specific; reference added. – Martijn Pieters Jun 09 '15 at 18:07

Why does Python 3 output \xe3, an extra char?

1 Answers1