Django UnicodeDecodeError after migrating to python 3.6

Question

I am migrating project from django 1.11 to django 2 and have the following problem. After migration from python 2.7 to python 3.6 on django 1.11 there are a lot of UnicodeDecodeError's when im trying to get some data by the ORM. For example:

class JournalName(models.Model):

    id = models.AutoField(primary_key=True)
    par_id = models.ForeignKey('Journal', db_column='par_id')
    parent = models.CharField(max_length=255)
    name = models.TextField(blank=True, db_column=u'name')
    lang = models.CharField(max_length=10, blank=True)

    class Meta:
        db_table = 'journal_name'

    def __str__(self):
        return self.name

following print :

print(JournalName.objects.all())

results with error:

UnicodeEncodeError: 'ascii' codec can't encode character '\u0142' in position 768: ordinal not in range(128)

Project is using MySQL database with utf8. I was trying to debug this but i have no idea what could cause the problem. On python 2.7 everything works ok so maybe there is something left in code that i should change compared to py2? I had to delete some option in db settings which caused other errors but maybe it has something with this:

'OPTIONS': {
   "init_command": "set names latin1;",
}

see answer here https://stackoverflow.com/questions/9942594/unicodeencodeerror-ascii-codec-cant-encode-character-u-xa0-in-position-20 — Reed Jones, Feb 15 '21 at 19:59
Ok, but i cannot encode Django QuerySet. I think IT is related with settings or something — tookas92, Feb 15 '21 at 20:09
on your model definition, what does the '____str____' function look like? — Reed Jones, Feb 15 '21 at 21:17

score 0 · Answer 1 · answered Feb 20 '21 at 13:57

Maybe, you are using ASCII locale (e.g. 'C' or 'POSIX').

You can print non-ASCII string by:

Use UTF-8 locales (e.g. 'en_US.UTF-8', 'C.UTF-8', etc...)
Set `PYTHONIOENCODING=utf-8' environment variable. But note that this only changes the stdio encoding.

But I strongly recommend to upgrade Python 3.7+. Since Python 3.7:

Python tries to change the locale to C.UTF-8 (See PEP 538) automatically when you are using 'C' or 'POSIX' locale.
If there is no "C.UTF-8" locale, Python enables UTF-8 mode automatically (See PEP 540).

So you can use UTF-8 more easily.

Django UnicodeDecodeError after migrating to python 3.6

1 Answers1