Some readble content, but impossible to JSON dump to file

Question

This text file (30 bytes only, the content is '(Ne pas r\xe9pondre a ce message)') can be opened and inserted in a dict successfully :

import json

d = {}

with open('temp.txt', 'r') as f:
   d['blah'] = f.read()

with open('test.txt', 'w') as f:
    data = json.dumps(d)
    f.write(data)    

But it is impossible to dump the dict into a JSON file (see traceback below). Why?

I tried lots of solutions provided by various SO questions. The closest solution I could get was this answer. When using this, I can dump to file, but then the JSON file looks like this:

# test.txt
{"blah": "(Ne pas r\u00e9pondre a ce message)"}

instead of

# test.txt
{"blah": "(Ne pas répondre a ce message)"}

Traceback:

  File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 9: invalid continuation byte
[Finished in 0.1s with exit code 1]

Answer 1

Your file is not UTF-8 encoded. It uses a Latin codec, like ISO-8859-1 or Windows Codepage 1252. Reading the file gives you the encoded text.

JSON however requires Unicode text. Python 2 has a separate Unicode type, and byte strings (type str) need to be decoded using a suitable codec. The json.dumps() function uses UTF-8 by default; UTF-8 is a widely used codec for encoding Unicode text data that can handle all codepoints in the standard, and is also the default codec for JSON strings to use (JSON requires documents to be encoding in one of 3 UTF codecs).

You need to either decode the string manually or tell json.dumps() what codec to use for the byte string:

data = json.dumps(d, encoding='latin1')  # applies to all bytestrings in d

or

d['blah'] = d['blah'].decode('latin1')
data = json.dumps(d)

or using io.open() to decode as you read:

import io

with io.open('test.txt', 'w', encoding='latin1') as f:
    d['blah'] = f.read()

By default, the json library produces ASCII-safe JSON output by using the \uhhhh escape syntax the JSON standard allows for. This is entirely normal, the output is valid JSON and readable by any compliant JSON decoder.

If you must produce UTF-8 encoded output without the \uhhhh escape sequences, see Saving utf-8 texts in json.dumps as UTF8, not as \u escape sequence