Django can't process non-ascii symbols

Question

We have a strange issue. If we use non-ascii symbols in Django, it displays them normally in the lists,

but if we will try to enter that item, it falls down with following error:

The sys.setdefaultencoding("utf-8") helps, but there is an opinion that this is a wrong way (Why should we NOT use sys.setdefaultencoding("utf-8") in a py script?). How can I solve it correctly? And why the lists displays properly? Any ideas appreciated!

Full traceback:

Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/enter/event/23/change/

Django Version: 1.9.4
Python Version: 2.7.10
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'debug_toolbar',
 'enter']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'debug_toolbar.middleware.DebugToolbarMiddleware']


Template error:
In template /Library/Python/2.7/site-packages/django/contrib/admin/templates/admin/change_form.html, error at line 21
   ascii   11 : {% block coltype %}colM{% endblock %}
   12 : 
   13 : {% block bodyclass %}{{ block.super }} app-{{ opts.app_label }} model-{{ opts.model_name }} change-form{% endblock %}
   14 : 
   15 : {% if not is_popup %}
   16 : {% block breadcrumbs %}
   17 : <div class="breadcrumbs">
   18 : <a href="{% url 'admin:index' %}">{% trans 'Home' %}</a>
   19 : &rsaquo; <a href="{% url 'admin:app_list' app_label=opts.app_label %}">{{ opts.app_config.verbose_name }}</a>
   20 : &rsaquo; {% if has_change_permission %}<a href="{% url opts|admin_urlname:'changelist' %}">{{ opts.verbose_name_plural|capfirst }}</a>{% else %}{{ opts.verbose_name_plural|capfirst }}{% endif %}
   21 : &rsaquo; {% if add %}{% trans 'Add' %} {{ opts.verbose_name }}{% else %} {{ original|truncatewords:"18" }} {% endif %}
   22 : </div>
   23 : {% endblock %}
   24 : {% endif %}
   25 : 
   26 : {% block content %}<div id="content-main">
   27 : {% block object-tools %}
   28 : {% if change %}{% if not is_popup %}
   29 :   <ul class="object-tools">
   30 :     {% block object-tools-items %}
   31 :     <li>


Traceback:

File "/Library/Python/2.7/site-packages/django/core/handlers/base.py" in get_response
  174.                     response = self.process_exception_by_middleware(e, request)

File "/Library/Python/2.7/site-packages/django/core/handlers/base.py" in get_response
  172.                     response = response.render()

File "/Library/Python/2.7/site-packages/django/template/response.py" in render
  160.             self.content = self.rendered_content

File "/Library/Python/2.7/site-packages/django/template/response.py" in rendered_content
  137.         content = template.render(context, self._request)

File "/Library/Python/2.7/site-packages/django/template/backends/django.py" in render
  95.             return self.template.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  206.                     return self._render(context)

File "/Library/Python/2.7/site-packages/django/test/utils.py" in instrumented_test_render
  92.     return self.nodelist.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Library/Python/2.7/site-packages/django/template/loader_tags.py" in render
  173.         return compiled_parent._render(context)

File "/Library/Python/2.7/site-packages/django/test/utils.py" in instrumented_test_render
  92.     return self.nodelist.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Library/Python/2.7/site-packages/django/template/loader_tags.py" in render
  173.         return compiled_parent._render(context)

File "/Library/Python/2.7/site-packages/django/test/utils.py" in instrumented_test_render
  92.     return self.nodelist.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Library/Python/2.7/site-packages/django/template/defaulttags.py" in render
  326.                 return nodelist.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Library/Python/2.7/site-packages/django/template/loader_tags.py" in render
  69.                 result = block.nodelist.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Library/Python/2.7/site-packages/django/template/defaulttags.py" in render
  326.                 return nodelist.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in render
  1043.             output = self.filter_expression.resolve(context)

File "/Library/Python/2.7/site-packages/django/template/base.py" in resolve
  736.                 new_obj = func(obj, *arg_vals)

File "/Library/Python/2.7/site-packages/django/template/defaultfilters.py" in _dec
  47.             args[0] = force_text(args[0])

File "/Library/Python/2.7/site-packages/django/utils/encoding.py" in force_text
  80.                 s = six.text_type(bytes(s), encoding, errors)

Exception Type: UnicodeEncodeError at /enter/event/23/change/
Exception Value: 'ascii' codec can't encode character u'\xe9' in position 29: ordinal not in range(128)

Please post the full traceback (as text). Also check your `settings.DEFAULT_CHARSET` and your database encoding (tables and client). — bruno desthuilliers, Aug 03 '18 at 14:08
Have you tried to encode them in `utf-8` directly instead ? (eg. `title.encode("utf-8")`) — scharette, Aug 03 '18 at 14:15
What's your `settings.DEFAULT_CHARSET` ? Do you have accented characters in your model's fields labels or help_text ? Do you have this problem on some records only or on all records ? Do you have the problem if trying to create a new event from the admin ? — bruno desthuilliers, Aug 03 '18 at 14:16
@scharette django uses the "unicode sandwich" pattern, so the template expects unicode strings and takes care of encoding them. — bruno desthuilliers, Aug 03 '18 at 14:19
@brunodesthuilliers **settings.DEFAULT_CHARSET** Didn't have this property in settings, but just tried to add it, and it has no effect **Do you have accented characters in your model's fields labels or help_text ?** In model's fields **Do you have this problem on some records only or on all records ?** It reproduces in all records with non-ascii symbols **Do you have the problem if trying to create a new event from the admin ?** Actually yes. Just checked it one more time right now — Arkady, Aug 03 '18 at 14:22
@brunodesthuilliers Replacing def __str__(self) with def __unicode__(self) in that model helps. What do you think? Is this correct fix?) — Arkady, Aug 03 '18 at 14:38
@Arkady can't tell without checking your code - there are many factors at play here, including whether you use `from __future__ import unicode_literals` or not and/or if you correctly use the `u""` unicode prefix , if you have `# -*- coding: ` headers in your source code, how your `__unicode__` method is implemented and whatnots... wrt/ the `settings.DEFAULT_CHARSET` is t defaults to utf-8 if not set, and since your error message makes clear mention of the ascii codec it's indeed not the culprit. — bruno desthuilliers, Aug 03 '18 at 15:00
As a general rule: if you want to be safe, you want to always only use unicode in your code, and specially for literal strings. Django itself converts everything to unicode at system entry points (HTTp queries, database queries etc). — bruno desthuilliers, Aug 03 '18 at 15:16

Arkady · Accepted Answer · 2018-08-07T07:57:54.743

0

In my case, replacing __ str__ method with __ unicode__ helped, but there are many factors at play here, as @bruno-desthuilliers said

# def __str__(self):
#     return str(self.id) +': ' + self.title

def __unicode__(self):
    return u'%s: %s' % (self.id, self.title)

edited Aug 07 '18 at 07:57

answered Aug 03 '18 at 15:08

Arkady

1,178
14
35

2

You DEFINITLY don't want to return a `str` from `__unicode__()` ! It has to be a unicode string. The proper implemenation here would be `return u"{}: {}".format(self.id, self.title)` – bruno desthuilliers Aug 03 '18 at 15:16

Django can't process non-ascii symbols

1 Answers1