Django charset and encoding

Question

I am trying to store and view greek characters from/to a Mysql db.

I want to use utf8 encoding i alter my db with the following command:

ALTER DATABASE el CHARACTER SET utf8 COLLATE utf8_general_ci

Now when i am trying to create an article i see the following error:

error page

When i am trying to fetch the article from my db i see something like:

mysql> select * from article_article
    -> ;
+----+------------+----------+-------------+----------------------------------------------------------------------------------+----------+------+-------+-------+--------------+---------------------+---------------------+-----------+
| id | submenu_id | title    | url         | body                                                                             | image_id | hits | votes | grade | publisher_id | time_upload         | last_modified       | published |
+----+------------+----------+-------------+----------------------------------------------------------------------------------+----------+------+-------+-------+--------------+---------------------+---------------------+-----------+
|  1 |          1 | ???????? | ssss-d-ssss | <p>&epsilon;&lambda;&lambda;&eta;&nu;&iota;&kappa;?</p>
<p>&nbsp;</p>
<p>?</p> |        9 |    0 |     0 |   2.5 |            1 | 2013-04-11 10:39:30 | 2013-04-11 11:02:01 |         1 |
+----+------------+----------+-------------+----------------------------------------------------------------------------------+----------+------+-------+-------+--------------+---------------------+---------------------+-----------+
1 row in set (0.00 sec)

I use tinymce for bodyfield..(Tinymce save greek characters very bad)

I also check that in admin page something like the following is missing:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

How can i configure django for utf8 format? How can i configure tinymce for Emding Greek Chars normally?

Edit: changing the collate of the field with mysql commands solve the problem. Also changing entity_encoding from named to raw in tiny_mce.js solve and the second problem.The only problem now is that the warning doesn't disapear :(

gitaarik · Accepted Answer · 2022-06-19T13:50:16.333

Set the charset option to utf8mb4 in the OPTIONS dict of the DATABASES setting in the Django settings file:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'my_db',
        'USER': 'my_user',
        'PASSWORD': 'my_pass',
        'HOST': 'my.host',
        'OPTIONS': {
            'charset': 'utf8mb4'  # This is the relevant line
        }
    }
}

These options will be passed as arguments to the mysql driver connection function, which typically is mysqlclient. Here you can see in the documentation all these parameters:

https://mysqlclient.readthedocs.io/user_guide.html#functions-and-attributes

If you use a different driver for mysql though, you might need a different option.

Also make sure that in the MySQL database, the charset of the either the column, the table or the whole database is set to utf8mb4. For more info about character sets in MySQL see:

MySQL docs: Specifying Character Sets and Collations
MySQL docs: The utf8mb4 Character Set (4-Byte UTF-8 Unicode Encoding)
Stack Overflow: What is the difference between utf8mb4 and utf8 charsets in MySQL?

There BTW an ongoing pull-request in Django to make utf8mb4 the default: https://github.com/django/django/pull/8886

If you get errors while changing MySQL Collation to utf8mb4, switch the storage engine to InnoDB. In my case it was MyISAM and the collation update failed with error "#1071 - Specified key was too long; max key length is 1000 bytes". — addmoss, Mar 31 '21 at 11:59

score 2 · Answer 2 · edited Jun 11 '21 at 15:06

2

in the Django settings file:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'OPTIONS': {
            'init_command': "SET NAMES 'utf8mb4';SET sql_mode='STRICT_TRANS_TABLES';",
            'read_default_file': '/platform/auth/mysql.conf',
            'charset': 'utf8mb4',
        },
        'ATOMIC_REQUESTS': True,
    }
}

edited Jun 11 '21 at 15:06

Adrian Mole

49,934
160
51
83

answered Jun 11 '21 at 10:20

wakeya

21
3

score 0 · Answer 3 · edited Feb 01 '21 at 15:27

gitaarik's answer above is correct and worked for me after a bit of digging. I'm adding this for those who use a separate file in the options dict. Like this:

DATABASES = {
'default': {
    'ENGINE': 'django.db.backends.mysql',
    'OPTIONS': {
        'read_default_file': '/platform/auth/mysql.conf',
        'charset': 'utf8mb4',
    },
    'ATOMIC_REQUESTS': True,
}

This was my mysql.conf. It didn't work until I removed the default-character-set option.

database = 'xyz_platform'
user = 'xyz'
password = 'xyzpassword'
default-character-set = 'utf8mb4'

The correct mysql.conf is this:

database = 'xyz_platform'
user = 'xyz'
password = 'xyzpassword'

Django charset and encoding

3 Answers3

Linked