Fastest way to convert a dict's keys & values from `unicode` to `str`?

Question

I'm receiving a dict from one "layer" of code upon which some calculations/modifications are performed before passing it onto another "layer". The original dict's keys & "string" values are unicode, but the layer they're being passed onto only accepts str.

This is going to be called often, so I'd like to know what would be the fastest way to convert something like:

{ u'spam': u'eggs', u'foo': True, u'bar': { u'baz': 97 } }

...to:

{ 'spam': 'eggs', 'foo': True, 'bar': { 'baz': 97 } }

...bearing in mind the non-"string" values need to stay as their original type.

Any thoughts?

RichieHindle · Accepted Answer · 2013-07-07T22:47:26.117

156

DATA = { u'spam': u'eggs', u'foo': frozenset([u'Gah!']), u'bar': { u'baz': 97 },
         u'list': [u'list', (True, u'Maybe'), set([u'and', u'a', u'set', 1])]}

def convert(data):
    if isinstance(data, basestring):
        return str(data)
    elif isinstance(data, collections.Mapping):
        return dict(map(convert, data.iteritems()))
    elif isinstance(data, collections.Iterable):
        return type(data)(map(convert, data))
    else:
        return data

print DATA
print convert(DATA)
# Prints:
# {u'list': [u'list', (True, u'Maybe'), set([u'and', u'a', u'set', 1])], u'foo': frozenset([u'Gah!']), u'bar': {u'baz': 97}, u'spam': u'eggs'}
# {'bar': {'baz': 97}, 'foo': frozenset(['Gah!']), 'list': ['list', (True, 'Maybe'), set(['and', 'a', 'set', 1])], 'spam': 'eggs'}

Assumptions:

You've imported the collections module and can make use of the abstract base classes it provides
You're happy to convert using the default encoding (use data.encode('utf-8') rather than str(data) if you need an explicit encoding).

If you need to support other container types, hopefully it's obvious how to follow the pattern and add cases for them.

edited Jul 07 '13 at 22:47

answered Aug 10 '09 at 12:03

RichieHindle

272,464
47
358
399

And what would one do if some values are lists/sets/etc? – Phillip B Oldham Aug 10 '09 at 12:12
@Philip: Add cases for them. Answer updated, and then updated again for nesting containers within containers. – RichieHindle Aug 10 '09 at 12:27
1

you forgot tuple and frozenset, Richi – SilentGhost Aug 10 '09 at 12:31
@SilentGhost: Lucky for you I'm so anally retentive I can't just leave it alone. 8-) Answer updated. – RichieHindle Aug 10 '09 at 12:36
Maybe he omitted them :) But yeah tuples are pretty common. – Skurmedel Aug 10 '09 at 12:36
I tried use this decision and get a exception `RuntimeError: maximum recursion depth exceeded`. My dict was simplest: {'Login': 'zxczxc', 'Name': '\xd0\x98\xd0\xbc\xd1\x8f', 'Surname': '\xd0\x9f\xd0\xbe\xd0\xbb\xd1\x8c\xd0\xb7\xd0\xbe\xd0\xb2\xd0\xb0\xd1\x82\xd0\xb5'} – Abbasov Alexander Jul 07 '13 at 16:28
I fixed it. For non unicode string should use `if isinstance(data, basestring):` instead `if isinstance(data, unicode):` – Abbasov Alexander Jul 07 '13 at 16:31
3

Why do you use `type(data)(map(convert, data))` instead `map(convert, data)`? – Abbasov Alexander Jul 11 '13 at 23:29
4

@AbbasovAlexander: So that you get back the same type you put in - a tuple becomes a tuple, a list becomes a list, a set becomes a set, and so on. – RichieHindle Jul 12 '13 at 07:08
Do I need to be afraid of hitting python's maximum recursion depth? – Moberg Nov 14 '14 at 15:48
1

@Moberg: Only if your data structure is nested many hundreds of levels deep. – RichieHindle Nov 14 '14 at 17:02
in Python3, `if isinstance(data, basestring):` to `if isinstance(data, str):` – zx1986 Jul 30 '17 at 04:30
I got no `convert` module in Python 3 :-( – zx1986 Jul 30 '17 at 04:31

score 25 · Answer 2 · answered Aug 11 '11 at 14:18

25

I know I'm late on this one:

def convert_keys_to_string(dictionary):
    """Recursively converts dictionary keys to strings."""
    if not isinstance(dictionary, dict):
        return dictionary
    return dict((str(k), convert_keys_to_string(v)) 
        for k, v in dictionary.items())

answered Aug 11 '11 at 14:18

Germano

2,452
18
25

1

Yup, this seems like the correct way of doing it, the inline and other versions really aren't enough for real world scenarios. Too bad there is no reliable inline recursionless way to accomplish this. Or maybe there is based on python str(...) json conventions? – jayunit100 Sep 15 '12 at 00:23
1

This is my favorite, to only convert the keys, which is what I was looking for. Small typo: you need an additional () around the dict() argument that is returned. – ggll Jul 30 '13 at 12:45
The only problem with this solution is if your keys are NOT all strings (i.e. int type) – MrWonderful Mar 14 '16 at 19:11
@MrWonderful and why is that? I can't see any problem in calling ``str`` on an int – Germano Mar 15 '16 at 09:29
@Germano : Of course you can call str() on an int, but you get a str.... not an int anymore. So the type of the key would be changed from an int to a str, which is *more* than changing unicode to str -- the original question. – MrWonderful Mar 15 '16 at 23:56
@MrWonderful I agree about the original question. The conversion to a string of an int is perfectly consistent with the name of the method though ;) – Germano Mar 16 '16 at 15:57
This only does keys. – Scott Tesler Jul 31 '17 at 22:01
This fails to handle other nested iterable types. – BuvinJ May 30 '18 at 13:55

score 12 · Answer 3 · answered Jul 18 '11 at 03:39

12

If you wanted to do this inline and didn't need recursive descent, this might work:

DATA = { u'spam': u'eggs', u'foo': True, u'bar': { u'baz': 97 } }
print DATA
# "{ u'spam': u'eggs', u'foo': True, u'bar': { u'baz': 97 } }"

STRING_DATA = dict([(str(k), v) for k, v in data.items()])
print STRING_DATA
# "{ 'spam': 'eggs', 'foo': True, 'bar': { u'baz': 97 } }"

answered Jul 18 '11 at 03:39

Samuel Clay

1,252
2
13
24

4

For 2.7 and onwards this can be simplified as follows: `{ str(key):value for key,value in data.items() }` – AnjoMan May 13 '15 at 14:49

maxbellec · Answer 4 · 2017-01-29T13:32:19.423

4

for a non-nested dict (since the title does not mention that case, it might be interesting for other people)

{str(k): str(v) for k, v in my_dict.items()}

edited Jan 29 '17 at 13:32

answered Jul 20 '16 at 08:45

maxbellec

16,093
10
36
43

1

{str(k): str(v) for k, v in my_dict.items()} – yardstick17 Jan 28 '17 at 10:44
This helped convert my keys to strings which i needed to compare against my dataframe column – megamind Mar 13 '19 at 16:15

score 3 · Answer 5 · answered Aug 10 '09 at 11:59

3

def to_str(key, value):
    if isinstance(key, unicode):
        key = str(key)
    if isinstance(value, unicode):
        value = str(value)
    return key, value

pass key and value to it, and add recursion to your code to account for inner dictionary.

answered Aug 10 '09 at 11:59

SilentGhost

307,395
66
306
293

score 2 · Answer 6 · answered Apr 04 '17 at 15:34

2

To make it all inline (non-recursive):

{str(k):(str(v) if isinstance(v, unicode) else v) for k,v in my_dict.items()}

answered Apr 04 '17 at 15:34

Ben

21
1

score 1 · Answer 7 · answered Aug 04 '20 at 03:20

1

>>> d = {u"a": u"b", u"c": u"d"}
>>> d
{u'a': u'b', u'c': u'd'}
>>> import json
>>> import yaml
>>> d = {u"a": u"b", u"c": u"d"}
>>> yaml.safe_load(json.dumps(d))
{'a': 'b', 'c': 'd'}

answered Aug 04 '20 at 03:20

lyu.l

282
3
8

Coddy · Answer 8 · 2020-05-19T16:49:44.263

0

Just use print(*(dict.keys()))

The * can be used for unpacking containers e.g. lists. For more info on * check this SO answer.

edited May 19 '20 at 16:49

answered May 19 '20 at 02:32

Coddy

549
4
18

Although this code might solve the problem, a good answer should explain **what** the code does and **how** it helps. – BDL May 19 '20 at 15:54

Fastest way to convert a dict's keys & values from `unicode` to `str`?

8 Answers8

Linked

Related