Python chokes on non-ascii characters when output to a UNIX pipeline

Question

Here is the reduced content of my script:

print u"w\xa0p"

which will print "w" character, followed by a nonbreaking space, then "p" to the TTY. However when this script is passed on to a pipeline, e.g.

python script.py | cat

(the exact command on the right of the pipeline is not important, as long as the python command is passed onto a UNIX pipeline), python will complain with this kind of error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 1: ordinal not in range(128)

Can somebody explain why this error happened, and how to mitigate it? Apparently there is a different codec for I guess we could use unidecode package to mitigate the issue at the cost of character precision; but I wonder if we can force python to print the utf-8 characters out anyway if the output pipeline is in use.

Answer 1

Have a look to this question, is the same problem you have there UnicodeDecodeError when redirecting to file

You should read all the answer from EOL to understand the problem, but if you are in a rush, here is the important part of the code you need to use:

import codecs
import locale
import sys

# Wrap sys.stdout into a StreamWriter to allow writing unicode.
sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout)

uni = u"w\xa0p"
print uni