overwrite python3 default encoder when using apache server

Question

I am running an apache server which serves a framework called ingenious

Getting UnicodeDecodeError('ascii' when reading a file with hebrew chars.

I've read that you can change the default preferred encoding of python3 using environment variables.

So I have edited the /etc/httpd/conf/httpd.conf using the [setenv][3] method:

SetEnv LC_ALL en_US.UTF-8
SetEnv LANG en_US.UTF-8
SetEnv LANGUAGE en_US.UTF-8
SetEnv PYTHONIOENCODING utf8

And restart the server using sudo service httpd restart and still not working.

I have to state that the software works locally when it's not running with an apache server, simply python3.

locale.getpreferredencoding() is ANSI_X3.4-1968 after the change

here is the content of /etc/httpd/conf/httpd.conf

ServerRoot "/etc/httpd"

Listen 80

Include conf.modules.d/*.conf

User apache
Group apache

ServerAdmin me@domain.com

<Directory />
    AllowOverride none
    Require all denied
</Directory>
ErrorLog "logs/error_log"

LogLevel warn

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    CustomLog "logs/access_log" combined
</IfModule>

<IfModule alias_module>
  ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>


<IfModule mime_module>
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>


AddDefaultCharset UTF-8

<IfModule mime_magic_module>

    MIMEMagicFile conf/magic
</IfModule>

EnableSendfile On

# Supplemental configuration

LoadModule wsgi_module /usr/lib64/python3.5/site-packages/mod_wsgi/server/mod_wsgi-py35.cpython-35m-x86_64-linux-gnu.so

WSGIScriptAlias / "/usr/bin/inginious-webapp.production"
WSGIScriptReloading On

Alias /static/common /usr/lib/python3.5/site-packages/inginious/frontend/common/static/
Alias /static/webapp /usr/lib/python3.5/site-packages/inginious/frontend/webapp/static/
Alias /static/lti /usr/lib/python3.5/site-packages/inginious/frontend/lti/static/

SetEnv LC_ALL en_US.UTF-8
SetEnv LANG en_US.UTF-8
SetEnv LANGUAGE en_US.UTF-8
SetEnv PYTHONIOENCODING utf8

<Directory "/usr/bin">
    <Files "inginious-webapp.production">
        Require all granted
    </Files>
</Directory>

<DirectoryMatch "/usr/lib/python3.5/site-packages/inginious/frontend/(.+)/static/">
    Require all granted
</DirectoryMatch>


IncludeOptional conf.d/*.conf

how can I further debug this?

score 2 · Accepted Answer · answered Oct 14 '17 at 22:22

2

Read:

http://blog.dscpl.com.au/2014/09/setting-lang-and-lcall-when-using.html

This explains issues around lang/locale.

You aren't using mod_wsgi daemon mode, but you should be as daemon mode is the recommended method.

Also go read the mod_wsgi documentation at:

http://www.modwsgi.org

answered Oct 14 '17 at 22:22

Graham Dumpleton

57,726
6
119
134

Thanks for the reply. I'm not in control of the devops part of this project, I'm afraid. but thanks for the tip regarding the mod_wsgi daemon – WebQube Oct 14 '17 at 22:45

Bernhard · Answer 2 · 2017-10-16T08:45:50.000

1

Python3 uses what local.getpreferredencoding returns as the encoding when opening files in text mode. I would suggest to add some logging and log the default encoding value to check whether it is the environment that's not configured properly.

It might be that there's a hard coded encoding parameter in the code somewhere. Can you show the stacktrace of the error and the corresponding code?

Depending on the deployment mechanism, it might be that the SetEnv configuration is not passed to Python. See http://ericplumb.com/blog/passing-apache-environment-variables-to-django-via-mod_wsgi.html for mod_wsgi deployments.

Similar to what is describe in that blog post, you have to modify the inginious-webapp.production file to get and set env vars manually. This is already done in the default inginious-webapp for some variables. Here you'd have to add in particular the PYTHONIOENCODING.

If you're running in mod_wsgi deamon mode, this might help http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html, in particular the lang or locale parameters.

edited Oct 16 '17 at 08:45

answered Oct 13 '17 at 14:59

Bernhard

8,583
4
41
42

locale.getpreferredencoding() is 'ANSI_X3.4-1968' after the change . updated the question. hard coded values - I doubt it since it's running fine in my local machine – WebQube Oct 13 '17 at 15:10
Obviously the env settings don't make it to Python's context. What deployment method are you using, mod_wsgi? – Bernhard Oct 13 '17 at 18:18
How do I know that ? Which deployment ? – WebQube Oct 13 '17 at 19:17
Well, because locale.getpreferredencoding does not return utf-8 ;) Deployment is how Apache invokes your Python application, there are several mechnisms, CGI, WSGI, WSGI in deamon mode, fastcgi.... I'm updating the answer with some hints. – Bernhard Oct 13 '17 at 20:02
hi, I've updated the question with the content of my /etc/httpd/conf/httpd.conf file. I don't how to make from that whether I'm running a WSGI or WSGI deamon.. – WebQube Oct 13 '17 at 20:28
I've update the answer with instructions to edit inginious-webapp.production. If that's not possible I can't help any further. – Bernhard Oct 16 '17 at 08:47

overwrite python3 default encoder when using apache server

2 Answers2

Linked