Serializing a nested namedtuple into JSON with Python >= 2.7

Question

I have a problem similar to CalvinKrishy's problem Samplebias's solution is not working with the data I have.

I am using Python 2.7.

Here's the data:

Namedtuple

>>> a_t = namedtuple('a','f1 words')
>>> word_t = namedtuple('word','f2 value')
>>> w1 = word_t(f2=[0,1,2], value='abc')
>>> w2 = word_t(f2=[3,4], value='def')
>>> a1 = a_t(f1=[0,1,2,3,4],words=[w1, w2])
>>> a1
a(f1=[0, 1, 2, 3, 4], words=[word(f2=[0, 1, 2], value='abc'), word(f2=[3, 4], value='def')])

Dict

>>> w3 = {}
>>> w3['f2'] = [0,1,2]
>>> w3['value'] = 'abc'
>>> w4 = {}
>>> w4['f2'] = [3,4]
>>> w4['value'] = 'def'
>>> a2 = {}
>>> a2['f1'] = [0, 1, 2, 3, 4]
>>> a2['words'] = [w3,w4]
>>> a2
{'f1': [0, 1, 2, 3, 4], 'words': [{'f2': [0, 1, 2], 'value': 'abc'}, {'f2': [3, 4], 'value': 'def'}]}

As you can see that both a1 and a2 are same except that one is namedtuple and other is dict.

But the json.dumps is different:

>>> json.dumps(a1._asdict())
'{"f1": [0, 1, 2, 3, 4], "words": [[[0, 1, 2], "abc"], [[3, 4], "def"]]}'
>>> json.dumps(a2)
'{"f1": [0, 1, 2, 3, 4], "words": [{"f2": [0, 1, 2], "value": "abc"}, {"f2": [3, 4], "value": "def"}]}'

I want to have json format of a1 exactly like its doing for a2.

A namedtuple is a subclass of tuple, so JSON serialises those as lists, as it should. — Martijn Pieters, Jun 05 '13 at 11:17
but http://docs.python.org/dev/library/collections.html#collections.somenamedtuple._asdict returns an OrderedDict — Kaushik Acharya, Jun 05 '13 at 11:38
That is a utility method; convert your tuples to ordered dicts then first before serialising to JSON. — Martijn Pieters, Jun 05 '13 at 11:51
You could create custom subclass of the encoder to handle named tuples: [Overriding nested JSON encoding of inherited default supported objects like dict, list](http://stackoverflow.com/q/16361223) — Martijn Pieters, Jun 05 '13 at 12:29
@MartijnPieters you can't override the json encoder's behavior in this case, as it sees namedtuples as tuples. — Emil, Mar 10 '14 at 13:47
@Emil: see the linked post, it deals with just that; overriding the default behaviour. — Martijn Pieters, Mar 10 '14 at 13:50
I suggest to [use a decorator](https://stackoverflow.com/questions/5906831/serializing-a-python-namedtuple-to-json/52224472#52224472). — Dmitry T., Sep 07 '18 at 14:25

score 13 · Answer 1 · answered Sep 25 '15 at 13:06

The problem is in the use of namedtuple._asdict, not json.dumps. If you look at the code with namedtuple(..., verbose=True) you will see this:

def _asdict(self):
    'Return a new OrderedDict which maps field names to their values'
    return OrderedDict(zip(self._fields, self))

Only the top level is actually changed into an OrderedDict, all contained elements are left untouched. This means that nested namedtuples still are tuple subclasses and get (correctly) serialized and such.

If the call to a specific conversion function is acceptable for you (like the call to _asdict), you can write your own one.

def namedtuple_asdict(obj):
  if hasattr(obj, "_asdict"): # detect namedtuple
    return OrderedDict(zip(obj._fields, (namedtuple_asdict(item) for item in obj)))
  elif isinstance(obj, basestring): # iterables - strings
     return obj
  elif hasattr(obj, "keys"): # iterables - mapping
     return OrderedDict(zip(obj.keys(), (namedtuple_asdict(item) for item in obj.values())))
  elif hasattr(obj, "__iter__"): # iterables - sequence
     return type(obj)((namedtuple_asdict(item) for item in obj))
  else: # non-iterable cannot contain namedtuples
    return obj

json.dumps(namedtuple_asdict(a1))
# prints '{"f1": [0, 1, 2, 3, 4], "words": [{"f2": [0, 1, 2], "value": "abc"}, {"f2": [3, 4], "value": "def"}]}'

As you can see, the biggest problem is having nested structures which are not namedtuples but could contain them.

Just a bit of warning: this function is not safe for any circular data structures. — MisterMiyagi, Mar 22 '16 at 07:08
For Python 3 replace `basestring` with `str` in the snippet. — Amit Kotlovski, Dec 04 '17 at 13:35

score 2 · Answer 2 · answered Oct 25 '18 at 13:02

Here's the version I went with, adapted from MisterMiyagi's. I used isinstance with collections.abc instead of hasattr, and I incuded a _type key in the resulting dict with the name of the namedtuple class.

import collections.abc

def _nt_to_dict(obj):
    recurse = lambda x: map(_nt_to_dict, x)
    obj_is = lambda x: isinstance(obj, x)
    if obj_is(tuple) and hasattr(obj, '_fields'):  # namedtuple
        fields = zip(obj._fields, recurse(obj))
        class_name = obj.__class__.__name__
        return dict(fields, **{'_type': class_name})
    elif obj_is(collections.abc.Mapping):
        return type(obj)(zip(obj.keys(), recurse(obj.values())))
    elif obj_is(collections.abc.Iterable) and not obj_is(str):
        return type(obj)(recurse(obj))
    else:
        return obj

Serializing a nested namedtuple into JSON with Python >= 2.7

Namedtuple

Dict

2 Answers2

Linked