python response.json preserve order

Question

I have a json object which is returned against a post call

r = requests.post("url", json=data)

I was doing r.json() to get the json object. But as I understand it creates a dict object which is unordered. I need to preserve the order.

I saw the solution described here: Items in JSON object are out of order using "json.dumps"?

But my challenge is my starting point is a response object. How do I take it and convert to a json with the order preserved?

Adding some more details:

My API call returns an object of form:

[{
    "key01": "value01",
    "key02": "value02",
    "keyN": "valueN"
},
{
    "key01": "value01",
    "key02": "value02",
    "keyN": "valueN"
},
{
    "key01": "value01",
    "key02": "value02",
    "keyN": "valueN"
}
]

I have a table with three columns as key01,key02 and keyN.

I need to post this json object after some minor manipulations to a software maintaining that specific order of key01,key02 and keyN.

But as soon as I do response.json() it is changing the order. I have tried to use the orderedlist approach as mentioned in the two other threads but so far my object is looking like this:

b"OrderedDict([('key01','value01'),('key02','value02'),('keyN','valueN')])

How do I get a json which looks like this instead: {"key01":"value01","key02":"value02","keyN":"valueN"}

Answer 1

The requests doesn't have the best documentation, but from reading its source code on the .json() method, we can see that it's defined as followed:

def json(self, **kwargs):
    r"""Returns the json-encoded content of a response, if any.
    :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
    :raises ValueError: If the response body does not contain valid json.
    """

    if not self.encoding and self.content and len(self.content) > 3:
        # No encoding set. JSON RFC 4627 section 3 states we should expect
        # UTF-8, -16 or -32. Detect which one to use; If the detection or
        # decoding fails, fall back to `self.text` (using chardet to make
        # a best guess).
        encoding = guess_json_utf(self.content)
        if encoding is not None:
            try:
                return complexjson.loads(
                    self.content.decode(encoding), **kwargs
                )
            except UnicodeDecodeError:
                # Wrong UTF codec detected; usually because it's not UTF-8
                # but some other 8-bit codec.  This is an RFC violation,
                # and the server didn't bother to tell us what codec *was*
                # used.
                pass
    return complexjson.loads(self.text, **kwargs)

where complexjson is the std json library or simplejson if you have that installed.

Knowing that, you can actually pass in keyword arguments to .json() which will directly go to json.loads(). Which means that you can do what the answer you linked proposed:

---

from collections import OrderedDict
r.json(object_pairs_hook=OrderedDict)

---

json.loads()

object_pairs_hook is an optional function that will be called with the result of any object literal decoded with an ordered list of pairs. The return value of object_pairs_hook will be used instead of the dict. This feature can be used to implement custom decoders. If object_hook is also defined, the object_pairs_hook takes priority.

simplejson.loads()

object_pairs_hook is an optional function that will be called with the result of any object literal decode with an ordered list of pairs. The return value of object_pairs_hook will be used instead of the dict. This feature can be used to implement custom decoders that rely on the order that the key and value pairs are decoded (for example, collections.OrderedDict will remember the order of insertion). If object_hook is also defined, the object_pairs_hook takes priority.

Therefore, either way you will be able to provide the object_pairs_hook keyword argument to r.json().

---

From the information I received from the comments, you don't even need to parse the json, just do:

text = r.content.decode(requests.utils.guess_json_utf(r.content)).encode('utf-8')

and you can "post" text to wherever you desire.

Answer 2

Relying on the order of json keys from a server (especially one that you don't control) is very fragile. The RFC says:

An object is an unordered collection of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array.

It also specifically comments:

JSON parsing libraries have been observed to differ as to whether or not they make the ordering of object members visible to calling software. Implementations whose behavior does not depend on member ordering will be interoperable in the sense that they will not be affected by these differences.

Thus, it is RFC compliant for an implementation (on the server) to change how it orders the produced JSON.

If you don't know if the server is using a serialization library that guarantees order, then this could break in the future (if the library changes). Even if you do, if that library takes the server language's equivalent of a dict, upgrading the language or standard library could change the semantics of that dict such that the ordering changes (and your code breaks). As an example, from Python 3.6 to 3.7 dicts changed from arbitrary order to ordered by insertion. In other languages such as rust, which seed the hash function used by their hashmaps to prevent DoS attacks, the ordering could be dependent on the randomness used to seed these hash functions (decided at runtime, and may be different if you, say, restart the server).

It's much more safe if you know that you need the data in a certain order to construct it that way yourself:

from collections import OrderedDict

ORDERED_KEYS = ['first', 'second', 'third']
ordered_json = OrderedDict((k, r.json()[k]) for k in ORDERED_KEYS)

From your comment, it seems like you need that dictionary serialized again. If you use json.dumps on an OrderedDict, the serialization will be in insertion order:

import json

serialized_ordered_json = json.dumps(ordered_json)