Django database charset issue

Question

I have the following issue for which i haven't get proper solution after an hour of searching.

I have a MySQL database table which has 'Long Text' column. To use less space for storing content of files in that text column the following compression approach has been used in PHP to store the content.

$compressed_content = bzcompress($content);
$db_compressed_content = addslashes($compressed_content);

The 'db_compressed_content' is stored in database using PHP itself.

Now i am in a position to utilize the database contents using Django. I was able to come up with the model class to represent the table. 'TextField' is used to represent that particular column.

Here is my exact issue, i used 'bz2.decompress()' of python to decompress and to get the text content but am getting 'UnicodeEncodeError' under django when i tried to do that.

FYI, the charset used to store content in database using PHP was 'latin-1'.

Please move the answer out of the question and into a proper answer below, this way you can mark the answer as answered and assist others looking for the solution — tutuDajuju, Feb 22 '14 at 07:11

score 6 · Answer 1 · answered Feb 25 '14 at 18:14

Answer: Django uses 'utf-8' as the default charset for the database, so if your database uses any other charset (mostly legacy database would be configured with 'latin1') then the charset need to be explicitly mentioned under database settings. Settings Ex:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'test_db',
        'USER': 'root',
        'PASSWORD': '',
        'HOST': '',
        'PORT': '3306',
        'OPTIONS': {
                    'charset': 'latin1',
                    'use_unicode': True, },
    }, 
}

In addition to this, if you don't need to use unicode then you can set the 'use_unicode' to False but i guess that is not recommended.

Cheers !!!

Django database charset issue

1 Answers1