UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 460: invalid continuation byte

Question

I've got a problem with the command loaddata of my Django app.

When I run this command, this UnicodeDecodeError appears because of the "é" that are in my database table.

i've try with many type of database on pgAdmin with different encode type like "UTF-8", "latin-1" or "win1252" or different "character type" or "collation" settings like "C" or "french_switzerland.1252" but every time there is this error.

Here is more information about my work space.

Let me know if you need more informations, I don't give you too much, I don't want to give useless information.

I know there are other questions which look similar to mine but I don't understand the answers. There are explanations about why there is the error but not about how to solve it.

I am on Windows 10
I'm using Pycharm 2020.3.2
Pgadmin v4

the complete error :

(venv) C:\Users\Mathias\PycharmProjects\yufindProject>python manage.py loaddata ask/dumps/ask.json
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()

  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\__init__.py", line 395, in execut
    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 172, in load_label
    for obj in objects:

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\serializers\json.py", line 67, in Deserializer
    stream_or_string = stream_or_string.decode()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 460: invalid continuation byte

Does this answer your question? [UnicodeDecodeError, invalid continuation byte](https://stackoverflow.com/questions/5552555/unicodedecodeerror-invalid-continuation-byte) — Piotr P. Karwasz, Jan 23 '21 at 13:31
no i don't see any solution, he just explain what is the error — Mathias.r, Jan 23 '21 at 13:59
Can you share some of the surrounding bytes `print(open('ask/dumps/ask.json', 'rb').read()[450:470])` — snakecharmerb, Jan 23 '21 at 15:02
`manage.py` assumes that `ask.json` is encoded in `UTF-8`, which clearly isn't the case. No database query was performed. Transcode the file to `UTF-8` and everything should work. — Piotr P. Karwasz, Jan 23 '21 at 15:25
@snakecharmerb i tried your command but it return me unvalid commutator - /dumps — Mathias.r, Jan 23 '21 at 18:35
@PiotrP.Karwasz how do i transcode file to utf-8 ? and which one, the json file ? sorry about my knowledge, i'm beginner ^^ — Mathias.r, Jan 23 '21 at 18:38
On Unixes you can use the `iconv` utility, but on any system you just open it with a good editor (VIM, Notepad++, ...) and change the encoding to UTF-8. — Piotr P. Karwasz, Jan 23 '21 at 18:44
@PiotrP.Karwasz okay but the encoding of which file ? the json file ? — Mathias.r, Jan 23 '21 at 19:02
@sorry i had bad understand your post, i'm gonna do this with ask.json thank you — Mathias.r, Jan 23 '21 at 19:11

score 0 · Answer 1 · answered Aug 19 '21 at 05:54

Found the solution in Django dumpdata fails on special characters

To save json data in django the TextIOWrapper is used:

The default encoding is now locale.getpreferredencoding(False) (...)

In documentation of locale.getpreferredencoding fuction we can read:

Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess.

Here I found "hacky" but working method to overwrite these settings:

In file settings.py of your django project add these lines:
import _locale
_locale._getdefaultlocale = (lambda *args: ['en_US', 'utf8'])

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 460: invalid continuation byte

1 Answers1