-1

I just ported my webapp to python 3. I develop in my Mac and deploy in a CentOS server. I found many UnicodeDecodeError that don't happen in my local test environment but they appear in the deployment server (of course :D )

Most of them I fixed by specifiying the encoding when opening files. However there is one place where I don't know how to specify encoding and it is in logging. I still get errors such as:

UnicodeEncodeError: 'ascii' codec can't encode character '\xab' in position 85: ordinal not in range(128)

The same problem existed (in both platforms) in python 2 and it was solved with this

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

Which changed the value of sys.getdefaultencoder() from ascii to utf-8

But now in python3 sys.getdefaultencoder() is already utf-8 by default ( Why should we NOT use sys.setdefaultencoding("utf-8") in a py script? ) so I'm clueless on what's causing this difference of behavior.

So:

  • what should I look for to see why both platforms are having different defaults for encoding?
  • how can I solve this for logging?
Martin Massera
  • 1,718
  • 1
  • 21
  • 47

1 Answers1

0

I found the answer here Python3 UnicodeDecodeError. Let me expand:

This is solved by setting the environment value LC_CTYPE to en_US.UTF-8 instead of UTF-8. This can be set in .bashrc:

export LC_CTYPE=en_US.UTF-8

Strangely enough, both my mac and deployment server have LC_CTYPE=UTF-8 and in my mac it just works, but in the deployment server I need to set it to en_US.UTF-8 otherwise it won't work.

But this seems like a weird config from my deployment server because if I set it to UTF-8 it complains like this:

$ export LC_CTYPE=UTF-8
bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

(My mac doesnt complain).

So obviously python is not reading LC_CTYPE directly but rather reading something else (a locale?) that is set by setting LC_CTYPE.

Martin Massera
  • 1,718
  • 1
  • 21
  • 47