7

I am unable to to dump collections.namedtuple as correct JSON.

First, consider the official example for using custom JSON serializer:

import json

class ComplexEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, complex):
            return [obj.real, obj.imag]
        # Let the base class default method raise the TypeError
        return json.JSONEncoder.default(self, obj)

json.dumps(2 + 1j, cls=ComplexEncoder)   # works great, without a doubt

Second, now consider the following example which tells Python how to JSONize a Friend object:

import json

class Friend():
    """ struct-like, for storing state details of a friend """
    def __init__(self, _id, f_name, l_name):
        self._id = _id
        self.f_name = f_name
        self.l_name = l_name

t = Friend(21, 'Steve', 'Rogerson')

class FriendEncoder(json.JSONEncoder):
    """ take a Friend object and make it truly json """
    def default(self, aFriend):
        if isinstance(aFriend, Friend):
            return {
                "id": aFriend._id,
                "f_name": aFriend.f_name,
                "l_name": aFriend.l_name,
            }
        return super(FriendEncoder, self).default(aFriend)

json.dumps(t, cls=FriendEncoder) # returns correctly JSONized string

Finally when we try to implement the same thing using namedtuples, json.dumps(t, cls=FriendEncoder) doesn't give any errors but gives the wrong output. Take a look:

import pdb
import json
from collections import namedtuple

Friend = namedtuple("Friend", ["id", 'f_name', 'l_name'])

t = Friend(21, 'Steve', 'Rogerson')

print(t)

class FriendEncoder(json.JSONEncoder):
    """ take a Friend collections.namedtuple object and make it truly json """
    def default(self, obj):
        if True:    # if isinstance(obj, Friend):
            ans = dict(obj._asdict())
            pdb.set_trace()     # WOW!! even after commenting out the if and hardcoding True, debugger doesn't get called
            return ans
        return json.JSONEncoder.default(self, obj)

json.dumps(t, cls=FriendEncoder)

The output I get is not a dict-like but rather just a list of values i.e. [21, 'Steve', 'Rogerson']

Why?

Is the default behavior such that information is lost?
Does json.dumps ignore the explicitly passed encoder?

Edit: by correctly jsonized namedtuple I mean that json.dumps should return data like exactly dict(nt._asdict()), where nt is a pre defined namedtuple

RinkyPinku
  • 410
  • 3
  • 20
  • 2
    The `default` method doesn't get called for subclasses of `tuple`, `list` etc. – vaultah Jul 16 '17 at 11:59
  • @vaultah I agree that `default` method isn't getting called. But can you substantiate "for subclasses of `tuple`, `list`, etc." – RinkyPinku Jul 16 '17 at 12:07
  • The `json.JSONEncoder` only calls `default ()` on objects it doesn't already know how serialize itself, which includes `tuple`s, `list`s and `dict`s (and subclasses of them). This means that in order to do what you want, you'll need to "trick" it. – martineau Jul 16 '17 at 13:44
  • Please describe you what think/want a `namedtuple` that has been "truly" converted to JSON would look like since such as thing is not a defined [JSON object](http://json.org/) Also, how do you expect it to be decoded later if it's non-standard? – martineau Jul 16 '17 at 13:56
  • "`json.JSONEncoder` only calls `default ()` on objects it doesn't already know how serialize itself, which includes `tuples`, `lists` and ..." I can't understand the underlying source code (`json/encoder.py`) can you please explain? – RinkyPinku Jul 16 '17 at 14:36
  • The `__repr__` of an object is just a string, which is also one of the thing sthe `json.JSONEncoder` already knows how to encode. The [**Encoders and Decoders**](https://docs.python.org/3/library/json.html#encoders-and-decoders) section of the `json` documentation has a list of them. I soon will post an answer that shows one way to get around this for specific types, like `namedtuple`s. – martineau Jul 16 '17 at 14:46

1 Answers1

9

As I said in a comment, the json.JSONEncoder only calls default when it encounters an object type it doesn't already know how to serialize itself. There's a table of them in the json documentation. Here's a screenshot of it for easy reference:

table of types supported by default

Note that tuple is on the list, and since namedtuple is a subclasses of tuple, it applies to them, too. (i.e. because isinstance(friend_instance, tuple) → True).

This is why your code for handling instances of the Friend class never gets called.

Below is one workaround — namely by creating a simple Wrapper class whose instances won't be a type that the json.JSONEncoder thinks it already knows how to handle, and then specifying a default= keyword argument function that's to be called whenever an object is encountered that it doesn't already know how to do.

Here's what I mean:

import json
from collections import namedtuple

class Wrapper(object):
    """ Container class for objects with an _asdict() method. """
    def __init__(self, obj):
        assert hasattr(obj, '_asdict'), 'Cannot wrap object with no _asdict method'
        self.obj = obj


if __name__ == '__main__':

    Friend = namedtuple("Friend", ["id", 'f_name', 'l_name'])
    t = Friend(21, 'Steve', 'Rogerson')
    print(t)
    print(json.dumps(t))
    print(json.dumps(Wrapper(t), default=lambda wrapped: wrapped.obj._asdict()))

Output:

Friend(id=21, f_name='Steve', l_name='Rogerson')
[21, "Steve", "Rogerson"]
{"id": 21, "f_name": "Steve", "l_name": "Rogerson"}

For some additional information and insights, also check out my answer to the related question Making object JSON serializable with regular encoder.

martineau
  • 119,623
  • 25
  • 170
  • 301