python: behavior of json.dumps on dict

Question

I am trying to override‎ the behavior of the dict on json.dumps. For instance, I can order the keys. Thus, I create a class which inherits if dict, and override‎ some of its methods.

import json

class A(dict):
    def __iter__(self):
        for i in range(10):
            yield i
    def __getitem__(self, name):
        return None

print json.dumps(A())

But it does not call any of my methods and only gives me {}

There is a way to give me the rigt behavior:

import json

class A(dict):
    def __init__(self):
        dict.__init__(self, {None:None})
    def __iter__(self):
        for i in range(10):
            yield i
    def __getitem__(self, name):
        return None

print json.dumps(A())

Wich finally gives {"0": null, "1": null, "2": null, "3": null, "4": null, "5": null, "6": null, "7": null, "8": null, "9": null}

Thus, it is clear that the C implementation of json.dumps somehow test if the dict is empty. Unfortunately, I cannot figure out which method is called. First, __getattribute__ does not work, and second I've overrided quite every method dict defines or could define without success.

So, could someone explain to me how the C implementation of json.dumps check if the dict is empty, and is there a way to override it (I find my __init__ pretty ugly).

Thank you.

Edit:

I finally found where this appends in the C code, and it looks not customizable

_json.c line 2083:

if (open_dict == NULL || close_dict == NULL || empty_dict == NULL) {
    open_dict = PyString_InternFromString("{");
    close_dict = PyString_InternFromString("}");
    empty_dict = PyString_InternFromString("{}");
    if (open_dict == NULL || close_dict == NULL || empty_dict == NULL)
        return -1;
}
if (Py_SIZE(dct) == 0)
    return PyList_Append(rval, empty_dict);

So it looks like Py_SIZE is used to check if the dict is empty. But this is a macro (not a function), which only return a property of the python object.

object.h line 114:

#define Py_REFCNT(ob)           (((PyObject*)(ob))->ob_refcnt)
#define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)
#define Py_SIZE(ob)             (((PyVarObject*)(ob))->ob_size)

So since its not a function, it cannot be overrided and thus its behavior cannot be customized.

Finally, the "non empty dict trick" is necessary if one want to customize json.dumps by inheriting a dict (of course other ways to achive this are possible).

Nope, does not use `__len__` neither. – Aurélien Lambert Dec 31 '13 at 11:13 — Aurélien Lambert, Dec 31 '13 at 11:13

aychedee · Answer 1 · 2013-12-31T10:32:17.433

2

Would it be easier to modify the behaviour of the encoder rather than creating a new dict sub class?

class OrderedDictJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if hasattr(obj, 'keys'):
            return {} # replace your unordered dict with an OrderedDict from collections
        else:
            return super(OrderedDictJSONEncoder, self).default(obj)

And use it like so:

json.dumps(my_dict_to_encode, cls=OrderedDictJSONEncoder)

This seems like the right place to turn an unordered Python dict into an ordered JSON object.

edited Dec 31 '13 at 10:32

answered Dec 31 '13 at 10:20

aychedee

24,871
8
79
83

Of course this would work, but I'd like it to be as transparent as possible. And since I already found a solution, the question is more "why" than "how". – Aurélien Lambert Dec 31 '13 at 11:15
The why is that to properly subclass the dict type you need to do something like this: http://stackoverflow.com/questions/3387691/python-how-to-perfectly-override-a-dict. In terms of transparency. I guess that's a matter of taste. I wouldn't want to use a special dict throughout a project if I could avoid it. – aychedee Dec 31 '13 at 13:16

score 1 · Answer 2 · edited May 23 '17 at 10:32

1

I don't know exactly what the encoder does, but it's not written in C, the Python source for the json package is here: http://hg.python.org/cpython/file/2a872126f4a1/Lib/json

Also if you just want to order the items, there's

json.dumps(A(), sort_keys=True)

Also see this question ("How to perfectly override a dict?") and its first answer, that explains that you should subclass collections.MutableMapping in most cases.

Or just give a subclassed encoder, as aychedee mentioned.

edited May 23 '17 at 10:32

Community

1
1

answered Dec 31 '13 at 10:28

RemcoGerlich

30,470
6
61
79

This is the correct answer! Thanks, I've never noticed the sort_keys option to dumps before. Duh! :) – aychedee Dec 31 '13 at 10:33
Yeah, but what if each embeded dictionnary needs a different order ? – Aurélien Lambert Dec 31 '13 at 11:16
1

encoder.py line 241: `_iterencode = c_make_encoder(markers, self.default, _encoder, self.indent, self.key_separator, self.item_separator, self.sort_keys, self.skipkeys, self.allow_nan)` – Aurélien Lambert Dec 31 '13 at 11:19

python: behavior of json.dumps on dict

2 Answers2