Django dumpdata fails on special characters

Question

I'm trying to dump my entire DB to a json. When I run python manage.py dumpdata > data.json I get an error:

(env) PS C:\dev\watch_something> python manage.py dumpdata > data.json
CommandError: Unable to serialize database: 'charmap' codec can't encode character '\u0130' in position 1: character maps to <undefined>
Exception ignored in: <generator object cursor_iter at 0x0460C140>
Traceback (most recent call last):
  File "C:\dev\watch_something\env\lib\site-packages\django\db\models\sql\compiler.py", line 1602, in cursor_iter
    cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.

It's because one of the characters in my DB is a sepcial character. How can I dump the DB correctly?

FYI, all other DB functionalities work fine

Julien Palard · Accepted Answer · 2021-06-12T06:09:57.767

72

One solution is to use ./manage.py dumpdata -o data.json instead of ./manage.py dumpdata > data.json.

Another solution is to use Python's UTF-8 mode, run:

python -Xutf8 ./manage.py dumpdata > data.json

edited Jun 12 '21 at 06:09

answered Apr 30 '21 at 15:28

Julien Palard

8,736
2
37
44

2

It really works, thanks a lot – Al Mahdi Sep 20 '21 at 03:54
works for me as well. Nice solution. – Yang Oct 23 '21 at 09:47
6

On Windows, using `python -Xutf8 manage.py dumpdata -o data.json` worked. Was failing without the `-Xutf8` option otherwise. – Supra621 Oct 28 '21 at 05:17
Yep! This worked Thanks! – Zack Plauché Feb 05 '22 at 13:22
The only clear solution – JM217 May 07 '22 at 16:38
what does -o do? – EinEsellesEniE Aug 03 '22 at 12:27
how can i add it back to the database? – Зуфар Тангрибергенов Nov 17 '22 at 17:34
Using `manage.py loaddata` (followed by the name of your file) (place your file in a `fixtures/` directory in your app) – Julien Palard Nov 17 '22 at 21:14

score 9 · Answer 2 · answered Dec 07 '20 at 18:00

To save json data in django the TextIOWrapper is used:

The default encoding is now locale.getpreferredencoding(False) (...)

In documentation of locale.getpreferredencoding fuction we can read:

Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess.

Here I found "hacky" but working method to overwrite these settings:

In file settings.py of your django project add these lines:

import _locale
_locale._getdefaultlocale = (lambda *args: ['en_US', 'utf8'])

If anyone should stumble upon this, please note that the syntax "python manage.py dumpdata > giveme.json" might still fail with this solution, but in my case "python manage.py dumpdata -o pleasegiveme.json" with the hacky solution above did work. — Morten, Feb 12 '21 at 09:24

score 4 · Answer 3 · edited Jul 15 '21 at 05:49

4

Here is the solution from djangoproject.com
You go to Settings there's a "Use Unicode UTF-8 for worldwide language support", box in "Language" - "Administrative Language Settings" - "Change system locale" - "Region Settings". If we apply that, and reboot, then we get a sensible, modern, default encoding from Python. djangoproject.com

the settings box looks like this. Enable the check and restart your system

edited Jul 15 '21 at 05:49

studioj

1,300
17
32

answered Feb 26 '21 at 14:54

Wertartem

237
2
5

1

This is now documented in the [official docs](https://docs.djangoproject.com/en/dev/howto/windows/#common-pitfalls) as well. – Supra621 Oct 28 '21 at 05:26

score 1 · Answer 4 · answered Aug 05 '22 at 02:17

On windows the way i solved mine was Add to your settings

import _locale
_locale._getdefaultlocale = (lambda *args: ['en_US', 'utf8'])

run this on shell only on windows

python -Xutf8 manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json

I actually did this it worked

python -Xutf8 manage.py dumpdata -o data.json

But wasnt showing data from my installed apps

score 0 · Answer 5 · answered May 08 '21 at 07:59

If you have multiple Python environments, before applying workarounds, it's worth checking that you issue python manage.py against the correct environment. That was my case where I met the same error: the database was created under containerized environment on Linux, with a higher Python version, but Django and other packages were present also in the legacy local environment on Windows. As well, the project directory was attached to the container as a volume, and the content was identical locally and in the container. So, I just mix up and ran manage.py locally instead of being attached to the container.

score 0 · Answer 6 · edited Mar 24 '23 at 09:28

I think this is the current best way: In the directory Python311\Lib\site-packages\django\core\serializers, you open the json.py file. In the json.py file, find the section:

class Serializer(PythonSerializer):  
           .................  
           def _init_options(self):  
                  .............  
                  # Default is False  
                  self.json_kwargs.setdefault("ensure_ascii", False)  
                  # You fix False to True  
                  self.json_kwargs.setdefault("ensure_ascii", True)

Django dumpdata fails on special characters

6 Answers6

Linked