Django utf-8 urls

Question

I have a Django app that works fine on localhost.even for utf-8 URL path.but when I use it in production it gives me an error:

2019-09-01 14:32:09.558237 [ERROR] [12257] wsgiAppHandler pApp->start_response() return NULL.
Traceback (most recent call last):
File "/home/medualla/virtualenv/project/3.7/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 139, in call 
set_script_prefix(get_script_name(environ))
File "/home/medualla/virtualenv/project/3.7/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 179, in get_script_name
script_url = get_bytes_from_wsgi(environ, 'SCRIPT_URL', '') or get_bytes_from_wsgi(environ, 'REDIRECT_URL', '')
File "/home/medualla/virtualenv/project/3.7/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 204, in get_bytes_from_wsgi
return value.encode('iso-8859-1')
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 1-6: ordinal not in range(256)

this error occurs when i try a url like
http://meduallameh.ir/صفحه
the only answer I got was that problem with the webserver. I deployed it on a shared host and I asked them and they told me that web server supports utf-8. now I need some help to fix this problem.

What's your server configuration? Are you running apache or nginx? gunicorn or uwsgi? — dirkgroten, Sep 02 '19 at 10:11
Note that browsers are sending this URI as `http://meduallameh.ir/%D8%B5%D9%81%D8%AD%D9%87`, HTTP only supports ASCII characters, therefore URLs get encoded. Your web server (e.g. apache) should just pass this encoded URL to your wsgi worker, so it's weird that this would be a problem. — dirkgroten, Sep 02 '19 at 10:20
@dirkgroten I told that webserver is LiteSpeed and I don't know configuration. I am using CPanel setup python app part. I send a ticket to host providers and they said that webserver is fine and many people have no problem with that. — Nimdeveloper, Sep 02 '19 at 10:41
@dirkgroten yeah im so confused that why this is happening. its change it to ASCII but Django throws an error — Nimdeveloper, Sep 02 '19 at 10:43
still would need to know what wsgi server is being use. LiteSpeed is just a web server. — dirkgroten, Sep 02 '19 at 10:55

score 3 · Accepted Answer · answered Sep 02 '19 at 14:05

3

After dealing with some code and searching for the problem I figured out that problem was that SCRIPT_URL and other stuff are decoded to utf-8 by default in the host. so it gives an error for that. I fixed it temporarily with changing get_bytes_from_wsgi return statement to this;

def get_bytes_from_wsgi(environ, key, default):
    """
    Get a value from the WSGI environ dictionary as bytes.

    key and default should be strings.
    """
    value = environ.get(key, default)
    # Non-ASCII values in the WSGI environ are arbitrarily decoded with
    # ISO-8859-1. This is wrong for Django websites where UTF-8 is the default.
    # Re-encode to recover the original bytestring.
    return value.encode('utf-8')

so the problem solved(for now). I figured out that this happens for many headers and especially with files. if someone finds another way that can fix, please write here

answered Sep 02 '19 at 14:05

Nimdeveloper

376
4
17

Dealing with a similar issue, but I'd rather not have to edit the Django library code. – jontsai Feb 17 '20 at 18:00
1

@jontsai I agree with you. if you have access to the web server, you can change the default encoding of the webserver to ISO-8859-1. if not, you have to contact the administrator. – Nimdeveloper Feb 17 '20 at 19:58
1

This is simply astounding. I'm trying to figure out how to submit a patch to Django. My webserver is Apache and proxying to Gunicorn for WSGI. I'm still trying to figure out where to configure my webserver to use ISO-8859-1, but patching the Django wsgi.py file as you have done works for now. – jontsai Feb 17 '20 at 21:17
1

I ended up creating a new file and subclassing WSGIHandler and WSGIRequest, and then pointed my WSGI entry point to my custom WSGI handler, rather than modifying Django library code. This way, I could deploy the solution to various servers/hosts reliably. Gist: https://gist.github.com/jontsai/afd5f5d9399ac2b0d770a73983d61690#file-django_wsgi_utf8_handler-py-L66-L69 – jontsai Feb 18 '20 at 04:52

score 2 · Answer 2 · answered Feb 18 '20 at 04:56

At one point, I had UTF-8 characters in the URL working for me, and somehow, I noticed that it broke when I migrated from Python 2 to Python 3 (along the way, I also switched from Apache + mod_wsgi to Apache proxying to Gunicorn).

Based on the answer the OP provided, I reluctantly resorted to subclassing the default WSGIHandler and WSGIRequest.

Gist: https://gist.github.com/jontsai/afd5f5d9399ac2b0d770a73983d61690#file-django_wsgi_utf8_handler-py-L66-L69

While not ideal, this solution can be deployed to multiple servers/environments.

I'll be looking to try to file a patch with the Django project.

Update Sept 11 2022: Somewhere between Django 3.0 and 3.2 and Ubuntu 20.04 -> Ubutnu 22.04, the above solution no longer works, and now in my `wsgi.py` file I simply go with: `from django.core.wsgi import get_wsgi_application; application = get_wsgi_application()` Instead of the previously custom subclassed WSGI Handler: `application = UTF8WSGIHandler()` — jontsai, Sep 12 '22 at 05:47

Django utf-8 urls

2 Answers2