Help me understand why Unicode only works sometimes with Python

Question

Here's a little program:

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

print('abcd kΩ ☠ °C √Hz µF ü ☃ ♥')  
print(u'abcd kΩ ☠ °C √Hz µF ü ☃ ♥')

On Ubuntu, Gnome terminal, IPython does what I would expect:

In [6]: run Unicodetest.py
abcd kΩ ☠ °C √Hz µF ü ☃ ♥
abcd kΩ ☠ °C √Hz µF ü ☃ ♥

I get the same output if I enter the commands on trypython.org.

codepad.org, on the other hand, produces an error for the second command:

abcd kΩ ☠ °C √Hz µF ü ☃ ♥
Traceback (most recent call last):
  Line 6, in <module>
    print(u'abcd kΩ ☠ °C √Hz µF ü ☃ ♥')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03a9' in position 6: ordinal not in range(128)

Contrariwise, IDLE on Windows mangles the output of the first command, but doesn't complain about the second:

>>>
abcd kΩ ☠ °C √Hz µF ü ☃ ♥
abcd kΩ ☠ °C √Hz µF ü ☃ ♥

IPython in a Windows command prompt or through Python(x,y)'s Console2 version both mangle the first output and complain about the second:

In [9]: run Unicodetest.py
abcd kΩ ☠ °C √Hz µF ü ☃ ♥
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (15, 0))

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)

Desktop\Unicodetest.py in <module>()
      4 print('abcd kΩ ☠ °C √Hz µF ü ☃ ♥')
      5
----> 6 print(u'abcd kΩ ☠ °C √Hz µF ü ☃ ♥')
      7
      8

C:\Python27\lib\encodings\cp437.pyc in encode(self, input, errors)
     10
     11     def encode(self,input,errors='strict'):
---> 12         return codecs.charmap_encode(input,errors,encoding_map)
     13
     14     def decode(self,input,errors='strict'):

UnicodeEncodeError: 'charmap' codec can't encode character u'\u2620' in position 8: character maps to <undefined>
WARNING: Failure executing file: <Unicodetest.py>

IPython inside Python(x,y)'s Spyder does the same, but differently:

In [8]: run Unicodetest.py
abcd kΩ ☠ °C √Hz µF ü ☃ ♥
------------------------------------------------------------
Traceback (most recent call last):
  File "Unicodetest.py", line 6, in <module>
    print(u'abcd kΩ ☠ °C √Hz µF ü ☃ ♥')
  File "C:\Python26\lib\encodings\cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u03a9' in position 6: character maps to <undefined>

WARNING: Failure executing file: <Unicodetest.py>

(In sitecustomize.py, Spyder sets its own SPYDER_ENCODING based on the locale module's encoding, which is cp1252 for Windows 7.)

What gives? Is one of my commands wrong? Why does one work on some platforms while the other works on other platforms? How do I print Unicode characters consistently without crashing or screwing up?

Is there an alternate terminal for Windows that behaves like the one in Ubuntu? It seems that TCC-LE, Console2, Git Bash, PyCmd, etc. are all just wrappers for cmd.exe rather than replacements. Is there a way to run IPython inside the interface that IDLE uses?

Answer 1

I/O in Python (and most other languages) is based on bytes. When you write a byte string (str in 2.x, bytes in 3.x) to a file, the bytes are simply written as-is. When you write a Unicode string (unicode in 2.x, str in 3.x) to a file, the data needs to be encoded to a byte sequence.

For a further explanation of this distinction see the Dive into Python 3 chapter on strings.

print('abcd kΩ ☠ °C √Hz µF ü ☃ ♥')

Here, the string is a byte string. Because the encoding of your source file is UTF-8, the bytes are

'abcd k\xce\xa9 \xe2\x98\xa0 \xc2\xb0C \xe2\x88\x9aHz \xc2\xb5F \xc3\xbc \xe2\x98\x83 \xe2\x99\xa5'

The print statement writes these bytes to the console as-is. But the Windows console interprets byte strings as being encoded in the "OEM" code page, which in the US is 437. So the string you actually see on your screen is

abcd kΩ ☠ °C √Hz µF ü ☃ ♥

On your Ubuntu system, this doesn't cause a problem because there the default console encoding is UTF-8, so you don't have the discrepancy between source file encoding and console encoding.

print(u'abcd kΩ ☠ °C √Hz µF ü ☃ ♥')

When printing a Unicode string, the string has to get encoded into bytes. But it only works if you have an encoding that supports those characters. And you don't.

• The default IBM437 encoding lacks the characters ☠☃♥
• The windows-1252 encoding used by Spyder lacks the characters Ω☠√☃♥.

So, in both cases, you get a UnicodeEncodeError trying to print the string.

What gives?

Windows and Linux took vastly different approaches to supporting Unicode.

Originally, they both worked pretty much the same way: Each locale has its own language-specific char-based encoding (the "ANSI code page" in Windows). Western languages used ISO-8859-1 or windows-1252, Russian used KOI8-R or windows-1251, etc.

When Windows NT added support for Unicode (int the early days when it was assumed that Unicode would use 16-bit characters), it did so by creating a parallel version of its API that used wchar_t instead of char. For example, the MessageBox function was split into the two functions:

int MessageBoxA(HWND hWnd, const char* lpText, const char* lpCaption, unsigned int uType);
int MessageBoxW(HWND hWnd, const wchar_t* lpText, const wchar_t* lpCaption, unsigned int uType);

The "W" functions are the "real" ones. The "A" functions exist for backwards compatibility with DOS-based Windows and mostly just convert their string arguments to UTF-16 and then call the corresponding "W" function.

In the Unix world (specifically, Plan 9), writing a whole new version of the POSIX API was seen as impractical, so Unicode support was approached in a different manner. The existing support for multi-byte encoding in CJK locales was used to implement a new encoding now known as UTF-8.

The preference towards UTF-8 on Unix-like systems and UTF-16 on Windows is a huge pain the the ass when writing cross-platform code that supports Unicode. Python tries to hide this from the programmer, but printing to the console is one of Joel's "leaky abstractions".

Answer 2

There are two possible reasons:

• Encoding of Unicode by print. You cannot output raw Unicode, so print needs to figure out how to convert it to the byte stream expected by the console (it uses sys.stdout.encoding AFAIK), which brings us to
• Console support. Python does not control your terminal, so if it spits out UTF-8 while your terminal expects something else, you'll get mangled output.

Answer 3

Your problem here is that your program expects, and outputs, UTF-8 characters, but consoles and various python runners on the web use other code pages. There is no way to code special characters that work in all encodings without modification. However, if you choose to use UTF-8 everywhere, you should be safe.

I think any terminal in Windows will do - so don't bother switching out the default one (cmd.exe) just because of this. Instead, change the encoding of the terminal to be UTF-8 as well, to match the encoding of your python script.

Unfortunately, I've never been able to find a way to set the code page to UTF-8 as default, so it has to be done every time you open a new command prompt. But it's done via a simple command, so it's only half-bad... You change the encoding by switching codepage:

>chcp 65001
Current codepage is now 65001

Note that you have to use one of the standard fonts for this to work. Most sources on the web seem to suggest Lucida Console.

Answer 4

Unicode output from Python to the Windows console just doesn't work. Python can't be persuaded to emit the native Windows encoding which expects wide characters and UCS2.

Answer 5

@dan04: You are right that the problem is that the encoding of the file does not match the encoding of stdout. Nevertheless one way to solve the problem is to change the encoding of the file. So on Windows Notepad++ can used to save the code with UTF-8 character encoding.

An alternative is GNU recode.