How to solve UnicodeDecodeError in Python 3.6?

Question

I am switched from Python 2.7 to Python 3.6.

I have scripts that deal with some non-English content.

I usually run scripts via Cron and also in Terminal.

I had UnicodeDecodeError in my Python 2.7 scripts and I solved by this.

# encoding=utf8  
import sys  

reload(sys)  
sys.setdefaultencoding('utf8')

Now in Python 3.6, it doesnt work. I have print statements like print("Here %s" % (myvar)) and it throws error. I can solve this issue by replacing it to myvar.encode("utf-8") but I don't want to write with each print statement.

I did PYTHONIOENCODING=utf-8 in my terminal and I have still that issue.

Is there a cleaner way to solve UnicodeDecodeError issue in Python 3.6?

is there any way to tell Python3 to print everything in utf-8? just like I did in Python2?

Are the non-English files encoded properly in UTF-8 themselves? — Edward Minnix, Jun 25 '18 at 14:52
@EdwardMinnix I am scraping data from various Hewbrew/Korean sites, so data is not always clean. — Umair Ayub, Jun 25 '18 at 14:53
@usr2564301 is there any way to tell Python3 to print everything in utf-8? just like I did in Python2? — Umair Ayub, Jun 25 '18 at 14:56
Normally your terminal has an encoding defined which is used by Python to set the encoding of its file object (`sys.stdout`). Can you provide what `sys.stdout.encoding` is set to on your machine? — Alfe, Jun 25 '18 at 14:57
@Alfe just checked, my Python2 and Python3 both has `ANSI_X3.4-1968` — Umair Ayub, Jun 25 '18 at 14:59
@usr2564301 so I will have to do `.encode("utf-8")` in every print statement — Umair Ayub, Jun 25 '18 at 14:59
I think that is the root of the problem. What strange terminal are you using? In Unix-ish environments you can set the env var `TERM` to something like `xterm` or similar. Also the `LANG` variable could have an influence. — Alfe, Jun 25 '18 at 15:00
@Alfe its a Ubuntu server and I am connected to it via MobaXterm — Umair Ayub, Jun 25 '18 at 15:00
Maybe this MobaXterm is configured strangely and can be improved. — Alfe, Jun 25 '18 at 15:07
Yet another reason for: https://stackoverflow.com/questions/3828723/why-should-we-not-use-sys-setdefaultencodingutf-8-in-a-py-script/34378962#34378962 — Alastair McCormack, Jun 25 '18 at 15:42
@Umair bingo - that's at least the start of your problem. Your operating system should be setting this for you. Please check your system is setup correctly — Alastair McCormack, Jun 25 '18 at 15:59
@AlastairMcCormack now I did `LANG=en_GB.UTF-8` and ran script, but still same error, (not a linux expert here) — Umair Ayub, Jun 25 '18 at 16:00
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/173761/discussion-between-alastair-mccormack-and-umair). — Alastair McCormack, Jun 25 '18 at 16:03

Alastair McCormack · Accepted Answer · 2018-06-25T15:51:51.613

It sounds like your locale is broken and have another bytes->Unicode issue. The thing you did for Python 2.7 is a hack that only masked the real problem (there's a reason why you have to reload sys to make it work).

To fix your locale, try typing locale from the command line. It should look something like:

LANG=en_GB.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_ALL=

locale depends on LANG being set properly. Python effectively uses locale to work out what encoding to use when writing to stdout in. If it can't work it out, it defaults to ASCII.

You should first attempt to fix your locale. If locale errors, make sure you've installed the correct language pack for your region.

If all else fails, you can always fix Python by setting PYTHONIOENCODING=UTF-8. This should be used as a last resort as you'll be masking problems once again.

If Python is still throwing an error after setting PYTHONIOENCODING then please update your question with the stacktrace. Chances are you've got an implied conversion going on.

For future readers, please see our conversation in chat how @Alastair McCormack helped me solve my problem. https://chat.stackoverflow.com/rooms/173761/discussion-between-alastair-mccormack-and-umair — Umair Ayub, Jun 25 '18 at 16:46
thanks for the answer and the discussion in the chat, It really helped me. I was facing a similar error in doing print('\u25cf') in python3. Setting the locale to en_US.utf8 helped. — code_dragon, Apr 03 '20 at 03:51
Thanks! When I type `locale` it tuned out that all values are "POSIX"... — Hardwired, Jun 09 '22 at 08:40

score 6 · Answer 2 · answered Sep 30 '19 at 19:14

6

I had this issue when using Python inside a Docker container based on Ubuntu 18.04. It appeared to be a locale issue, which was solved by adding the following to the Dockerfile:

ENV LANG C.UTF-8

answered Sep 30 '19 at 19:14

Daniel

473
4
9

2

`export LANG=en_GB.UTF-8` before running python in console fix this temporarily – zhy Sep 13 '21 at 15:42

score 4 · Answer 3 · answered Apr 15 '21 at 21:41

4

To everyone using pickle to load a file previously saved in python 2 and getting an UnicodeDecodeError, try setting pickle encoding parameter:

with open("./data.pkl", "rb") as data_file:
    samples = pickle.load(data_file, encoding='latin1')

answered Apr 15 '21 at 21:41

Mark Storm

51
2

Alfe · Answer 4 · 2018-06-25T15:24:29.470

1

For a Python-only solution you will have to recreate your sys.stdout object:

import sys, codecs
sys.stdout = codecs.getwriter('utf-8')(sys.stdout.detach())

After this, a normal print("hello world") should be encoded to UTF-8 automatically.

But you should try to find out why your terminal is set to such a strange encoding (which Python just tries to adopt to). Maybe your operating system is configured wrong somehow.

EDIT: In my tests unsetting the env variable LANG produced this strange setting for the stdout encoding for me:

LANG= python3
import sys
sys.stdout.encoding

printed 'ANSI_X3.4-1968'.

So I guess you might want to set your LANG to something like en_US.UTF-8. Your terminal program doesn't seem to do this.

edited Jun 25 '18 at 15:24

answered Jun 25 '18 at 15:05

Alfe

56,346
20
107
159

Is this the same or significantly different from `sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf8')`? (Which I am sure I have learned from this very website.) – Jongware Jun 25 '18 at 15:18
1

@usr2564301 I'm not sure about the difference (if any). The idea is clearly the same. A much nicer way would be the simple `sys.stdout.encoding = 'utf-8'` but that doesn't work, unfortunately, because the `encoding` field is readonly. But this is always the father of the thought and the two solutions are just different workaround implementations of it. – Alfe Jun 25 '18 at 15:22

score -1 · Answer 5 · answered Jun 25 '18 at 15:00

Python 3 (including 3.6) is already Unicode supported. Here is the doc - https://docs.python.org/3/howto/unicode.html

So you don't need to force Unicode support like Python 2.7. Try to run your code normally. If you get any error reading a Unicode text file you need to use the encoding='utf-8' parameter while reading the file.

score -1 · Answer 6 · answered May 08 '22 at 17:47

-1

for docker with python3.6, use LANG=C.UTF-8 python or jupyter xxx works for me, thanks to @Daniel and @zhy

answered May 08 '22 at 17:47

zhibo

29
5

score -3 · Answer 7 · answered Jun 25 '18 at 14:54

-3

I mean you could write an custom function like this: (Not optimal i know)

import sys

def printUTF8(input):
    print(input.encode("utf-8"))

answered Jun 25 '18 at 14:54

Jakob Sachs

669
4
24

any way without doing `.encode("utf-8")`? so Python knows to print every string in utf-8? – Umair Ayub Jun 25 '18 at 14:55

How to solve UnicodeDecodeError in Python 3.6?

7 Answers7

Linked