How to remove unicode characters from Dictionary data in python

Question

After using request library , am getting below dict in response.json()

{u'xyz': {u'key1': None, u'key2': u'Value2'}}

I want to remove all unicode characters and print only key value pairs without unicode chars

I have tried below method to remove , but it shows malformed string

>>> import json, ast
>>> c = {u'xyz': {u'key1': None,u'key2': u'Value2'}}
>>> ast.literal_eval(json.dumps(c))

Getting 'ValueError: malformed string'

Any suggestion on how to do it ?

What is the output you want, and why? – RemcoGerlich Jun 28 '16 at 11:19 — RemcoGerlich, Jun 28 '16 at 11:19
try using `json.dumps(c)` – Tony Roczz Jun 28 '16 at 11:49 — Tony Roczz, Jun 28 '16 at 11:49

S.K. Venkat · Answer 1 · 2018-08-27T17:40:32.213

This snippet will helps you to preserve the data without unicode prefix notation u :

>>> import json
>>> c = {u'xyz': {u'key1': u'Value1',u'key2': u'Value2'}}
>>> print c
{u'xyz': {u'key2': u'Value2', u'key1': u'Value1'}}
>>> d = eval(json.dumps(c))
>>> print d
{'xyz': {'key2': 'Value2', 'key1': 'Value1'}}

json.dumps() will convert the dict to string type and eval() will reverse it.

Note: key1 value has changed from None to 'value1' for testing purpose

score 4 · Accepted Answer · edited May 23 '17 at 12:32

4

Change your None to 'None':

 c = {u'xyz': {u'key1': 'None', u'key2': u'Value2'}}

it is a casting issue - ast likes str's

Also, maybe u want to change all None to empty str or 'None' str... See this thread : Python: most idiomatic way to convert None to empty string? with this code, i've changes the empty string to 'None':

def xstr(s):
    if s is None:
        return 'None'
    return str(s)

edited May 23 '17 at 12:32

Community

1
1

answered Jun 28 '16 at 11:15

Ohad the Lad

1,889
1
15
24

1

Then it becomes a string with "None" as the value, completely different from None. – RemcoGerlich Jun 28 '16 at 11:19

score 1 · Answer 3 · edited Sep 24 '21 at 14:56

I don't really understand why you want this. Your variable is a normal Python dict with normal Unicode strings, and they happen to be printed as u'' to distinguish them from bytestrings, but that shouldn't matter for using them.

If you want to save them as strings to read them as data later, JSON is a fine format for that. So no need to call request's .json() function at all, just use the response's .text attribute -- it's already JSON, after all.

Your try

>>> ast.literal_eval(json.dumps(c))

Fails because you first turn c into JSON again, and then try to parse it as a Python literal. That doesn't work because Python isn't JSON; in particular one has null and the other has None.

So maybe you want to change the Unicode strings into bytestrings? Like by encoding them as UTF8, that might work:

def to_utf8(d):
    if type(d) is dict:
        result = {}
        for key, value in d.items():
            result[to_utf8(key)] = to_utf8(value)
        return result
    elif type(d) is unicode:
        return d.encode('utf8')
    else:
        return d

Or something like that, but I don't know why you would need it.

score 1 · Answer 4 · answered Jun 28 '16 at 13:53

1

You can use unicodestring.encode("ascii","replace")

>>> ustr=u'apple'
>>> ustr
u'apple'
>>> astr=ustr.encode("ascii","replace")
>>> astr
'apple'

answered Jun 28 '16 at 13:53

Marichyasana

2,966
1
19
20

This for strings not for dict – Orkun Jan 14 '21 at 08:54

How to remove unicode characters from Dictionary data in python

4 Answers4