3

I have dictionary of dictionary which has utf8 encoded keys. I am dumping this dictionary to a file using json module.
In the file keys are printed in utf8 format. Keys are actually letters of Bengali language.

I want actual letters to get written in the file. How to do this ??

If I print those keys(one of them is u'\u0982') to console actual letter(ং) is shown but in my file \u0982.is written. What does print do to show the actual letter?

Abhishek Gupta
  • 6,465
  • 10
  • 50
  • 82
  • 1
    This talks about unicode reading in python, not sure if it what you are looking for http://stackoverflow.com/questions/491921/unicode-utf8-reading-and-writing-to-files-in-python. – debianplebian Jun 20 '13 at 13:25

2 Answers2

2

You are writing JSON; the JSON standard allows for \uxxxx escape sequences to encode non-ASCII characters. The Python json module uses this by default.

Switch off the feature by using the ensure_ascii=False switch when dumping the data:

json.dump(obj, yourfileobject, ensure_ascii=False)

This does mean that the output is no longer encoded to UTF-8 bytes as well; you'll need to use a codecs.open() managed file for this:

import json
import codecs

with codecs.open('/path/to/file', 'w', encoding='utf8') as output:
    json.dump(obj, output, ensure_ascii=False)

Now your unicode characters will be written to the file as UTF-8 encoded bytes instead. When opening the file with another program that decodes UTF-8 again, your codepoints should be displayed again as the same characters.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • But I am getting this `'ascii' codec can't encode character u'\u0982' in position 1: ordinal not in range(128)` error when dump the data with `ensure_ascii=False`. This error is generated for the first key which is `\u0982` when I dump with default value i.e True – Abhishek Gupta Jun 20 '13 at 13:36
  • @knoxxs: mea culpa, indeed the `json.dump()` output then produces Unicode values and does *not* encode. Updated to include a method to write the data and have it encoded for you. – Martijn Pieters Jun 20 '13 at 13:46
2

use ensure_ascii parameter.

>>> import json
>>> print json.dumps(u'\u0982')
"\u0982"
>>> print json.dumps(u'\u0982', ensure_ascii=False)
"ং"

http://docs.python.org/2/library/json.html#json.dump

If ensure_ascii is True (the default), all non-ASCII characters in the output are escaped with \uXXXX sequences, and the result is a str instance consisting of ASCII characters only. If ensure_ascii is False, some chunks written to fp may be unicode instances. ...

falsetru
  • 357,413
  • 63
  • 732
  • 636