1

I have a Django 1.8 app running on a server with Python 3 and I get a UnicodeDecodeError when logging and printing strings with special characters.

>:python --version python 3.4.3

For example, if I try to run a silly method in the shell:

def print_test():
       print('Test: èè') # any 'special char' like ä ç é û...

I get a stack trace:

>>> print_test()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/sailingadmin/sailing-admin/utest.py", line 2, in print_test
    print('This is a test: \xe8\xe8')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 16-17: ordinal not in range(128)

Same with print(u'Test: èè')

Why does this error happen?

utest.py is encoded in utf-8 (Python 3 default for source files)

>:echo $LC_CTYPE UTF-8

All logging and printing raises a UnicodeEncodeError...

Ire
  • 1,341
  • 10
  • 16
  • What's important here are the LC_* environment variable _used by your server process_ - not the ones defined for your own user account. You may want to check your process's environment and front-end server config. – bruno desthuilliers Nov 27 '15 at 10:39
  • @brunodesthuilliers Do you mean the `/proc//environ`? LC_CTYPE=UTF-8 and LANG=en_US.UTF-8, no other encoding variables are set there. – Ire Nov 27 '15 at 10:50
  • I mean "the environment variables of the process which stdout will be used by your process"... Sorry I've not had to deal with this kind of problem for quite a few years now (thanks to our sysadmin ), so I don't remember all the gory details :-/ – bruno desthuilliers Nov 27 '15 at 10:55
  • Take a look here, maybe that will help you set the encoding back to UTF-8: http://stackoverflow.com/questions/2276200/changing-default-encoding-of-python – Maciek Nov 27 '15 at 11:28
  • @Maciek this does not apply to python 3, reload is not a builtin. (but it is importable) – Ire Nov 27 '15 at 11:37
  • @Ire can you check what is your default locale while executing the script using locale.getlocale() and locale.getpreferredencoding()? And do you use the internationalization features of Django? – Maciek Nov 27 '15 at 11:43
  • @Maciek locale.getlocale() returns (None, None) and getprefferedencoding() returns 'ANSI_X3.4-1968' I am trying to find out where this value comes from! – Ire Nov 27 '15 at 11:50
  • LC_CTYPE=UTF-8 wasn't valid. Set it to en_US.UTF-8, and now the errors are gone! Thanks! – Ire Nov 27 '15 at 12:11
  • Add an answer to your question then and accept it after 48 hours :) – Maciek Nov 27 '15 at 12:22

1 Answers1

2

LC_CTYPE format as follows: en_US.UTF-8 (char map after the dot) Then python uses it as default encoding for logging and io.

Ire
  • 1,341
  • 10
  • 16