What is the most economical way to convert nested Python objects to dictionaries?

Question

I have some SQLAlchemy objects which contain lists of more SQLAlchemy objects, and so on (for about 5 levels). I wish to convert all the objects to dictionaries.

I can convert an object to a dictionary by using the __dict__ property, no problem. However, I'm having trouble figuring out the best way to convert all the nested objects as well, without having to do each level explicitly.

So far, this is the best I can come up with, but it doesn't recurse properly. It basically breaks after one pass, so there's clearly something wrong with my logic. Can you see what's wrong with it??

I am hoping to do:

all_dict = myDict(obj.__dict__)

def myDict(d):
    for k,v in d.items():
        if isinstance(v,list):
            d[k] = [myDict(i.__dict__) for i in v]
        else:
            d[k] = v
    return d

Do you want the resulting dictionaries to mimic the same nested structure, or do you want to flatten it? — Tim Pietzcker, Nov 01 '11 at 07:57
Note that by using the `__dict__` property, you will also include all the methods and stuff from base-classes. — Björn Pollex, Nov 01 '11 at 08:01
Tim, I would settle for either but my goal is to flatten using an existing key as a reference. Does that make sense? — MFB, Nov 01 '11 at 08:07
Björn, I was going to weed out the base class junk, but point me to a better way. make my own to_dict method? — MFB, Nov 01 '11 at 08:09
See this answer: http://stackoverflow.com/questions/1036409/recursively-convert-python-object-graph-to-dictionary/1118038#1118038 But watch out for self-referential data structures which it can't handle. — Björn Lindqvist, Nov 01 '11 at 09:26
Thanks, close but sadly I have backrefs in the alchemy objects so that method loops — MFB, Nov 01 '11 at 12:01
@bjorn-lindqvist please take a look at my code effort. I'd love to know your thoughts — MFB, Nov 01 '11 at 13:47

score 46 · Answer 1 · edited Jul 06 '22 at 18:32

46

Life Hack:

def to_dict(obj):
    return json.loads(json.dumps(obj, default=lambda o: o.__dict__))

This leverages the default input of the json.dumps() method (or json.dump()) by returning the __dict__ representation of the objects that are not serializable. Note that this may not work when objects contain non-standard data structures (such as NumPy or Pandas). The following is the description from the documentation of json.dump:

If specified, default should be a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError. If not specified, TypeError is raised.

edited Jul 06 '22 at 18:32

Steven C. Howell

16,902
15
72
97

answered Jan 24 '18 at 00:59

zachdb86

971
1
8
13

This is a much cleaner solution and appears to do the trick using Python 3 – Matt Haidet Jul 31 '20 at 15:42
I love this! I was thinking to implement this using recursion, but this simplifies the problem brilliantly! – Steven C. Howell Jul 06 '22 at 16:50
I had some nested python `dataclass`s and this function did the trick, thanks! – Emad Goahri Oct 25 '22 at 16:49

score 15 · Accepted Answer · edited Jul 04 '16 at 14:55

I am not sure if I understood exactly what you want - but if I got, this function can do what you want: It does search recursively on an object's attributes, yielding a nested dictionary + list structure, with the ending points being python objects not having a __dict__ attribute - which in SQLAlchemy's case are likely to be basic Python types like numbers and strings. (If that fails, replacing the "hasattr dict" test for soemthing more sensible should fix the code for your needs.

def my_dict(obj):
    if not  hasattr(obj,"__dict__"):
        return obj
    result = {}
    for key, val in obj.__dict__.items():
        if key.startswith("_"):
            continue
        element = []
        if isinstance(val, list):
            for item in val:
                element.append(my_dict(item))
        else:
            element = my_dict(val)
        result[key] = element
    return result

Thanks jsbueno. It works, I just had to modify it to exclude the self-referencing objects in my data. Cheers! — MFB, Nov 03 '11 at 02:45

What is the most economical way to convert nested Python objects to dictionaries?

2 Answers2

Linked