In place editing of Python dictionary or parsing json to object

Question

I am by no means a Python Guru, but I know my way around. For the past two hours however I've been banging my head against the following:

I am parsing a JSON response from a WCF Webservice using the json.loads() function. The result is a Python dictionary which I am using throughout my application. However, I now have the need to obfuscate the id, reseller_id etc. for use in HTTP GET requests.

This is an example of a response: (note, I have many of these responses, so I am looking for a generic solution.) I want to replace the value of any id with a hash of the id value

{
    "token":"String content",
    "user":{
        "distributor":{
            "email":"String content",
            "id":2147483647,
            "name":"String content"
        },
        "email":"String content",
        "first_name":"String content",
        "id":2147483647,
        "last_name":"String content",
        "reseller":{
            "email":"String content",
            "id":2147483647,
            "name":"String content",
            "portal_css":"String content",
            "portal_logo":"String content",
            "portal_name":"String content",
            "portal_url":"String content"
        },
        "role":2147483647
    }
}

I have tried all kinds of strategies using code like:

result = json.loads(json_result, object_hook=lambda d: namedtuple('X', d.keys())(*d.values()))

and

def fun(d):
    if 'id' in d:
        yield d['id']
    for k in d:
        if isinstance(d[k], list):
            for i in d[k]:
                for j in fun(i):
                    yield j

I can't get it to work properly. So:

Question 1: Can I convert json to (Anonymous) Python Objects and how?

Question 2: Can I edit the resulting dict in place?

Question 3: When all else fails; Does anybody have an idea how to achieve this?

Thanks a lot!

`json.loads(json_result)` gives you a dictionary. What do you mean by an "anonymous Python object"? Can you give an example of your desired output? (I think you want to hash the IDs, but I don't see any attempt to hash anything in your code samples). And what kind of hashing do you want? — David Robinson, Apr 16 '13 at 15:00
To clarify, do the "HTTP GET requests" you mention require passing the whole dict, or just the hashed ID? — Aya, Apr 16 '13 at 15:01
@Aya, Just the hashed id, like /reseller/28763298728/Reseller-A/users in stead of /reseller/1/users — Berdus, Apr 16 '13 at 15:04
TBH, you may as well just leave the dict as-is, and calculate the hash on-demand when you need it. — Aya, Apr 16 '13 at 15:05
@DavidRobinson, Believe you me, I tried, I use a hash function similar to http://stackoverflow.com/questions/4273466/reversible-hash-function By anonymous I mean A non existing class. When I use the namedtuple approach I get a "Can't Pickle type... attribute lookup failed" — Berdus, Apr 16 '13 at 15:08
@Aya, I could - and am for certain functions - but when using Ajax I run into the above problem. I can't "reach" the dict but in javascript. That's too late :( — Berdus, Apr 16 '13 at 15:10
@Berdus JavaScript? You don't mention that in the question at all. Perhaps you include that in the question. — Aya, Apr 16 '13 at 15:14

score 3 · Accepted Answer · answered Apr 16 '13 at 15:04

3

json.loads(data)

parses the json string data and returns a Python dict, which I guess you could call anonymous, until you assign it to a variable:

d = json.loads(data)

You can then modify the dict in place. Assuming that all of your responses have the same structure, you can hash the id as follows:

d["user"]["id"] = your_hash_function(d["user"]["id"])

answered Apr 16 '13 at 15:04

Elmar Peise

14,014
3
21
40

Is it? I tried this, but no satisfactory result. I will try again. I am still looking for a more generic approach, but if this works I'll gladly use your approach in my "WCFBackend" class functions. – Berdus Apr 16 '13 at 15:13
@Berdus where exactly is your problem? Do you get an error or is something not working as expected? – Elmar Peise Apr 16 '13 at 15:15
I'm a little embarrassed to say that *** result["user"]["id"] = "test" *** seems to work. In my quest I must have overlooked a bug in my code after wich I concluded that dicts are immutable. I am stil interested in a more generic approach though. Your answer however does fit the bill! – Berdus Apr 16 '13 at 15:27
@Berdus so is this giving an error (if so, which) or not yielding the expected result? – Elmar Peise Apr 16 '13 at 15:28

score 1 · Answer 2 · edited Apr 16 '13 at 15:13

1

This is a solution that I quickly put together. It recurses through your dictionary and replaces the value when the key is id. This isn't the final solution as it doesn't support lists, but I'm sure if this works for you, you can take it from there?

def f(d):
    _dict = {}
    for k, v in d.items():
        if isinstance(v, dict):
            _dict.update({k: f(v)})
        else:
            if k == 'id':
                print 'id found', v
                v = EncryptValueHere(v)
            _dict.update({k:v})

    return _dict

edited Apr 16 '13 at 15:13

Abgan

3,696
22
31

answered Apr 16 '13 at 15:11

Werner Smit

1,951
1
13
9

because the OP is interested in changing the values, cant you directly change the values in the original dict, instead of creating a new one? – GodMan Apr 16 '13 at 15:13
That would be possible yes. Dictionaries are passed by reference after all. Will look at reworking this to support that. Unless this dictionary is very huge it shouldn't be too much of a performance hit to replace the whole dictionary. – Werner Smit Apr 16 '13 at 15:22

score 1 · Answer 3 · answered Apr 16 '13 at 15:28

Yes. json.loads returns a Python object based on the actual json object. Check the conversion table for more details. There is no need to look for a special anonymous object like you might in .Net land.

result = json.loads(json_text)

You can edit this result in place:

result['user']['id'] = 'some_new_value'

Try the object_pairs_hook argument when parsing your json:

def hide_ids(o):
    d = dict(o)
    if 'id' in d:
        d['id'] = '----{}----'.format(d['id'])
    return d

result = json.loads(json_text, object_pairs_hook=hide_ids)

Just modify with your preferred obfuscation logic and you should be good to go.

Looks promising... Thanks for your comment, I accepted ExP's answer just now. +1 for the object_pairs_hook. PS: I have of course looked at the docs. In the Netherlands we have a saying (loosely translated): "Staring oneself blind". This probably lies at the root of my problem. — Berdus, Apr 16 '13 at 15:32

In place editing of Python dictionary or parsing json to object

3 Answers3