Can not encode a dict to JSON within a custom encoder

Question

I am trying to make a custom JSON encoder and I ran into this problem I don't understand.

I basically followed the instructions of the official documentation

Encoding a dictionary is clearly possible with the standard JSON encoder. But if I convert the custom object to a dict and pass it to the default encoder, I get an: TypeError: Object of type dict is not JSON serializable.

Clearly there is something I am missing here. Can someone explain to me the behavior of the toy example below?

import json

class Klass():
    def __init__(self,number):
        self.number = number

    def __dict__(self):
        return {"number": self.number}

class CustomEncoder(json.JSONEncoder):
    def default(self,obj):
        if isinstance(obj,Klass):
            obj=obj.__dict__()
        return json.JSONEncoder.default(self, obj)

json.dumps({"number" : 10})                         # works
json.dumps({"number" : 10},cls=json.JSONEncoder)    # works
json.dumps({"number" : 10},cls=CustomEncoder)       # works
json.dumps(Klass(10).__dict__(), cls=CustomEncoder) # works
json.dumps({"Test":Klass(10).__dict__()}, cls=CustomEncoder) #works

try:
    json.dumps(Klass(10), cls=CustomEncoder) # TypeError: Object of type dict is not JSON serializable
except TypeError:
    print("Error 1")
# this is my end goal to encode a dict of objects
try:
    json.dumps({"Test":Klass(10)}, cls=CustomEncoder) # TypeError: Object of type dict is not JSON serializable
except TypeError:
    print("Error 2")

# this works but clearly it shows me the Custom encoder is not doing what I think it does
encode_hack = {k: v.__dict__() for k, v in {"Test":Klass(10)}.items()}
json.dumps(encode_hack)

The `default()` method of a custom encoder should convert the object into some valid JSON representation (a string) and return that. — martineau, Nov 12 '21 at 13:10
`default` doesn't return the JSON representation; it returns a JSON-serializable value. `encode` returns a JSON representation, in part by using `default`. — chepner, Nov 12 '21 at 13:24
To be more precise, default() should return something encodable using one of the default encodings https://docs.python.org/3/library/json.html#encoders-and-decoders - for example in the docs ComplexEncoder returns a list. Also note that the encoder is extensible but not overrideable - i.e. you can't replace how those default encodings are done. — DisappointedByUnaccountableMod, Nov 12 '21 at 13:25
@balmy If you structure things correctly, the value returned by one class's `default` method can be something produced by another's. *Eventually*, the sequence of calls to `default` needs to produce the values that `json.JSONEncoder._iterencode` itself knows how to encode (that's where `None` is turned into `'null'`, for instance). — chepner, Nov 12 '21 at 13:27
If you make `Klass` inherit from `dict` then there's no need for a custom encoder, but everything in the `__dict__` of the Klass instance you're encoding will be in the result. — DisappointedByUnaccountableMod, Nov 12 '21 at 13:30

martineau · Accepted Answer · 2021-11-12T18:09:28.597

You shouldn't define a class method named __dict__, it's a special read-only built-in class attribute, not a method, and you don't need to overload in order to do what you want.

Here's a modified version of your code that show how to do things:

import json

class Klass:
    def __init__(self,number):
        self.number = number

# Don't do this.
#    def __dict__(self):
#        return {"number": self.number}

class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Klass):
#            obj=obj.__dict__()
             return {"number": obj.number}
        return json.JSONEncoder.default(self, obj)

json.dumps({"number" : 10})                         # works
json.dumps({"number" : 10}, cls=json.JSONEncoder)   # works
json.dumps({"number" : 10}, cls=CustomEncoder)      # works
json.dumps(Klass(10).__dict__, cls=CustomEncoder) # works
json.dumps({"Test":Klass(10).__dict__}, cls=CustomEncoder) #works

try:
    json.dumps(Klass(10), cls=CustomEncoder)
except TypeError as exc:
    print(exc)

# this is my end goal to encode a dict of objects
try:
    json.dumps({"Test":Klass(10)}, cls=CustomEncoder)
except TypeError as exc:
    print(exc)

# this works but clearly it shows me the Custom encoder is not doing what i think it does
encode_hack = {k: v.__dict__ for k, v in {"Test":Klass(10)}.items()}
json.dumps(encode_hack)

Update

An even better way in my opinion to do it would be to rename your __dict__() method to something not reserved and have your custom encoder call it. A major advantage being that now it's now more object-oriented and generic in the sense that any class with a method of that name could also be encoded (and you don't have to hardcode a class name like Klass in your encoder).

Here's what I mean:

import json

class Klass:
    def __init__(self,number):
        self.number = number

    def _to_json(self):
        return {"number": self.number}


class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        try:
            return obj._to_json()
        except AttributeError:
            return super().default(obj)

# A single object.
try:
    print(json.dumps(Klass(10), cls=CustomEncoder))  # -> {"number": 10}
except TypeError as exc:
    print(exc)

# A dict of them.
try:
    print(json.dumps({"Test":Klass(42)}, cls=CustomEncoder)) # -> {"Test": {"number": 42}}
except TypeError as exc:
    print(exc)

You *can* define `__dict__`, but it's a bad idea to do so :) — chepner, Nov 12 '21 at 13:28
Probably a matter of opinion, but I think it's cleaner to keep using an explicit method in `Klass` to define the dictionary that `CustomEncoder.default` will use; just don't name that method `__dict__`. — chepner, Nov 12 '21 at 13:42
@chepner: Yes, something like a `_to_json()` method. That way the custom encoder could be generalized to support any class that defined one (i.e. not just one like `Klass`). — martineau, Nov 12 '21 at 13:46
I sometimes wonder why the `json` module *didn't* take that route. — chepner, Nov 12 '21 at 13:50
@chepner: Hindsight's always 20/20. Fortunately it's easy to add to a custom encoder, but it would be nice to have an officially sanctioned name. — martineau, Nov 12 '21 at 13:56
After thinking about it, I suspect it's because the current approach doesn't require reserving *any* names outside of the encoder subclass itself. — chepner, Nov 12 '21 at 14:11
Thank you a lot. And as well quite interesting discussion here. — Erysol, Nov 15 '21 at 08:55
Erysol: In that case, you might also find [my answer](https://stackoverflow.com/a/18561055/355230) to the question [Making object JSON serializable with regular encoder](https://stackoverflow.com/questions/18478287/making-object-json-serializable-with-regular-encoder) of interest too. — martineau, Nov 15 '21 at 10:04

score 1 · Answer 2 · answered Nov 12 '21 at 13:44

Dunder names are reserved for use by the implementation. Don't reuse them outside their documented usage, and don't invent your own.

import json

class Klass():
    def __init__(self,number):
        self.number = number

    def to_dict(self):
        return {"number": self.number}


class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Klass):
            return obj.to_dict()
        return json.JSONEncoder.default(self, obj)

Can not encode a dict to JSON within a custom encoder

2 Answers2

Update