pycharm with codecs utf-8 encode from system stdout and stderr

Question

hellow all, I'm working with python 3.6.0 on a project in pycharm community edition 2016.3.2. I have this strange error happens in my program whenever I use codecs to encode my input stream:

import codecs
import sys

sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)

print("something")

the problem is my program always exit with an exit code 1, and here is the console output:

Traceback (most recent call last):
  File "C:/Users/jhonsong/Desktop/restfulAPI/findARestaurant.py", line 9, in <module>
    print("something")
  File "C:\Python36-32\lib\codecs.py", line 377, in write
    self.stream.write(data)
TypeError: write() argument must be str, not bytes

Process finished with exit code 1

but my input is a string, why is codecs.write think I'm giving it bytes input?

This sounds like a bad idea anyway - not all consoles support UTF-8. You should print and allow Python's locale detection to work out the appropriate encoding — Alastair McCormack, Jun 06 '17 at 21:02
@AlastairMcCormack what I'm doing is trying to get this online class challenge done: [link](https://classroom.udacity.com/courses/ud388/lessons/4628431284/concepts/52474372990923), it evolves with foursquare's API and possible multiple language results return. you are suggesting write the result to a file instead of console? — paradox, Jun 06 '17 at 21:04
See: https://stackoverflow.com/questions/4374455/how-to-set-sys-stdout-encoding-in-python-3 — Alastair McCormack, Jun 06 '17 at 21:08

Mark Tolonen · Answer 1 · 2017-06-09T01:15:19.627

In Python 3, sys.stdout is a Unicode stream. codecs.getwriter('utf8') needs a byte stream. sys.stdout.buffer is the raw byte stream, so use:

sys.stdout = codecs.getwriter('utf8')(sys.stdout.buffer)
sys.stderr = codecs.getwriter('utf8')(sys.stderr.buffer)

But this seems like overkill with Python 3.6, where the print should just work. Normally if you need to force an encoding, say to capture the output of a Python script to a file in a certain encoding, the following script (Windows) would work. The environment variable dictates the output encoding and overrides whatever the terminal default is.

set PYTHONIOENCODING=utf8
python some_script.py > output.txt

Since you are submitting to a website, that site should be responsible for running your script in a way that they can capture the result properly.

pycharm with codecs utf-8 encode from system stdout and stderr

1 Answers1