Convert tweepy JSON object to dict

Question

I'd like to convert the result of a Tweepy api.trends_location(woeid) call to a dict (or a dict of dicts), so I can work with the values (really, I want to end up with a dict of the 'name' values). The Tweepy documentation says that the result is 'a JSON object' (see here), but when I retrieve it, type(retrieved) evaluates to list. Sure enough, retrieved has a len of 1, and retrieved[0] gives me a single item:
[{'trends': [{'url': 'http://search.twitter.com/search?q=%23questionsidontlike', 'query': '%23questionsidontlike', 'events': None, 'promoted_content': None, 'name': '#questionsidontlike'}, ], (more of the same), 'created_at': '2011-01-31T22:39:16Z', 'as_of': '2011-01-31T22:47:47Z', 'locations': [{'woeid': 23424977, 'name': 'United States'}]}].

I can call json.dumps, which will give a nicely-formatted representation, but that's not much use to me, and json.loads gives me: __init__() got an unexpected keyword argument 'sort_keys'

How should I proceed?

Link to full code: https://gist.github.com/805129

JSON objects in Python are in fact represented using `list` and `dict` objects. `dict` is used for objects and `list` for arrays. What exactly are you trying to do ? — thkala, Jan 31 '11 at 23:21
I understand that, but a list with one member, which is the entire result isn't much use, in this case. I'd like to extract the values of 'name' into their own list. — urschrei, Jan 31 '11 at 23:28

Andrew Clark · Accepted Answer · 2011-02-01T00:49:53.733

3

Okay this should do it! It is even tested (thanks for posting the additional info).

>>> names = [trend["name"] for trend in retrieved[0]["trends"]]
>>> names
['#wishuwould', '#questionsidontlike', '#februarywish', 'Purp & Patron', 'Egyptians', 'Kool Herc', 'American Pie', 'Judge Vinson', 'Eureka Nutt', 'Eddie House']

I think most of the confusion came from the documentation referring to the output as a JSON object, which is different than a JSON string which would need to be converted using the json module.

How this works: retrieved is a list that contains a single item, which is the dictionary that contains the trends key, so retrieved[0]["trends"] is the list of trend dictionaries, where each trend dictionary contains the name key you are interested in.

edited Feb 01 '11 at 00:49

answered Jan 31 '11 at 23:39

Andrew Clark

202,379
35
273
306

I tried this quickly, but am only getting '0' as the output for retrieved[0][0], So something odd is going on. Have edited original with link to full code. – urschrei Feb 01 '11 at 00:18
Okay it looks like the output is still in a string format, I had thought it was already converted to Python objects. Try `json.loads(retrieved[0])` and see if that works, I'll start editing my comment. – Andrew Clark Feb 01 '11 at 00:24
Hmm, json.loads(retrieved[0]) is giving me a 'expected string or buffer' error, and json.loads(str(retrieved[0])) gives me 'Expecting property name: line 1 column 1 (char 1)' – urschrei Feb 01 '11 at 00:38
Well I guess my theory is wrong, could you post the entire output of `print retrieved`? Either here or on github. – Andrew Clark Feb 01 '11 at 00:39
OK, making some progress: type(retrieved[0]) is a dict. Here's the retrieved output (click raw to see it properly): https://gist.github.com/805173 – urschrei Feb 01 '11 at 00:44
Check my most recent edit, this should be what you're looking for. – Andrew Clark Feb 01 '11 at 00:50
Yep, I just finished testing with a less elegant slice and loop. Nice comprehension, too. Thanks! – urschrei Feb 01 '11 at 00:53

score 2 · Answer 2 · answered Jan 31 '11 at 23:39

Would something like this work for you?

def searchKeys(struct, keys, result = None, recursive = True):
        if result is None:
                result = []

        if isinstance(struct, dict):
                for k in keys:
                        if struct.has_key(k):
                                result.append(struct[k])

                if recursive:
                        for i in struct.values():
                                searchKeys(struct = i, keys = keys, result = result, recursive = recursive)
        elif isinstance(struct, list):
                if recursive:
                        for i in struct:
                                searchKeys(struct = i, keys = keys, result = result, recursive = recursive)

        return result

Usage example:

>>> searchKeys(struct = a, keys = ['name'])
['United States', '#questionsidontlike']

It recursively walks down a dict/list hierarchy searching for a set of dict keys and stores the corresponding values to a list.

score 1 · Answer 3 · answered Feb 11 '17 at 15:24

1

To convert a Tweepy 'Status' object to a Python dictionary (JSON), access the private member "_json" on the object.

tweets = tweepy_api.user_timeline(screen_name='seanharr11')
json_tweets = map(lambda t: t._json, tweets)

answered Feb 11 '17 at 15:24

The Aelfinn

13,649
2
54
45

score 0 · Answer 4 · answered Jan 31 '11 at 23:53

0

>>> import simplejson
>>> a = {"response":[{"message":"ok"},{"message":"fail"}]}
>>> json = simplejson.dumps(a)
>>> simplejson.loads(json)
{'response': [{'message': 'ok'}, {'message': 'fail'}]}

http://docs.python.org/library/json.html

answered Jan 31 '11 at 23:53

jbcurtin

1,793
2
14
23

Convert tweepy JSON object to dict

4 Answers4

Linked