Python string not printing properly in PowerShell

Question

I'm having difficulty parsing data with a lot of scientific and international symbols using Python 2.7 so I wrote a toy program that illustrates what is not making sense to me:

#!/usr/bin/python
# coding=utf-8
str ="35 μg/m3"
str = str.decode('utf-8') 
str = str.encode('utf-8') #ready for printing? 
print(str)

And instead of printing out the original content, I get something different:

Answer 1

The line # coding=utf-8 only helps to write unicode literal and is no use for plain byte strings. Anyway assuming that your Python file is UTF-8 encoded, the line str = str.decode('utf-8') gives you a correct unicode string.

But as said by Ansgar Wiechers, as you declare your encoding the simpler way would be to directly use a unicode litteral:

str = u"35 μg/m3"

Simply, Windows console has poor support for UTF8. Common encodings are win1252 (a latin1 variant), or cp850 a native OEM font. Unless you want to explicitely deal with the explicit encoding, your best bet is to directly display the unicode string:

#!/usr/bin/python
# coding=utf-8
str ="35 μg/m3"
str = str.decode('utf-8') # str is now an unicode string
print(str)

---

If you want to explicitely use latin1, and provided you use a TrueType font such as Lucida Console or Consolas, you can do:

chcp 1252
python .\encoding.py

with

#!/usr/bin/python
# coding=utf-8
str ="35 μg/m3"
str = str.decode('utf-8') # str is now an unicode string
str = str.encode('latin1') # str is now an latin1 encoded byte string
print(str)

Answer 2

Python 2.7 doesn't use Unicode strings by default, so you basically have 2 options:

• Define the string as a Unicode string literal (u"..."):

  # coding=utf-8
  str = u"35 µg/m3"
  print(str)
  

  This way you can simply use the string as one would expect, so I'd prefer this approach.

• Define the string as a regular string literal and decode it:

  # coding=utf-8
  str = "35 \xc2\xb5g/m3"
  print(str.decode('utf-8'))
  

  If you use this approach you need to put special characters as their hexadecimal values (µ in UTF-8 is the character sequence 0xC2,0xB5) even if the file is saved as UTF-8.

Demonstration:

PS C:\> $PSVersionTable.PSVersion.ToString()
4.0
PS C:\> C:\Python27\python.exe -V
Python 2.7.11
PS C:\> Get-Content .\test.py -Encoding UTF8
# coding=utf-8
str1 = "35 \xc2\xb5g/m3"
print(str1)
print(str1.decode('utf-8'))
str2 = u"35 µg/m3"
print(str2)
PS C:\> C:\Python27\python.exe .\test.py
35 ┬Ág/m3
35 µg/m3
35 µg/m3

Answer 3

Your decoding/encoding has no effect:

# coding=utf-8
s1 = "35 μg/m3"
s2 = s1.decode('utf-8') 
s3 = s2.encode('utf-8') #ready for printing?
print s1==s3

If your source is UTF-8 as declared, then s1 is a byte string that is UTF-8-encoded already. Decoding it to a Unicode string (s2) and re-encoding it as UTF-8 just gives you the original byte string.

Next, the Windows console does not default to UTF-8, so printing those bytes will intepret them in the console encoding, which on my system is:

import sys
print sys.stdout.encoding
print s3

Output:

cp437
35 ┬╡g/m3

The correct way to print Unicode strings and have them intepreted correctly is to actually print Unicode strings. They will be encoded to the console encoding by Python and display correctly (assuming the console font and encoding supports the characters).

# coding=utf-8
s = u"35 µg/m3"
print s

Output:

35 µg/m3