Python 3 Get and parse JSON API

Question

How would I parse a json api response with python? I currently have this:

import urllib.request
import json

url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'

def response(url):
    with urllib.request.urlopen(url) as response:
        return response.read()

res = response(url)
print(json.loads(res))

I'm getting this error: TypeError: the JSON object must be str, not 'bytes'

What is the pythonic way to deal with json apis?

score 63 · Accepted Answer · edited Apr 15 '19 at 18:37

Version 1: (do a pip install requests before running the script)

import requests
r = requests.get(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
print(r.json())

Version 2: (do a pip install wget before running the script)

import wget

fs = wget.download(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
with open(fs, 'r') as f:
    content = f.read()
print(content)

score 47 · Answer 2 · answered Dec 28 '16 at 21:07

you can use standard library python3:

import urllib.request
import json
url = 'http://www.reddit.com/r/all/top/.json'
req = urllib.request.Request(url)

##parsing response
r = urllib.request.urlopen(req).read()
cont = json.loads(r.decode('utf-8'))
counter = 0

##parcing json
for item in cont['data']['children']:
    counter += 1
    print("Title:", item['data']['title'], "\nComments:", item['data']['num_comments'])
    print("----")

##print formated
#print (json.dumps(cont, indent=4, sort_keys=True))
print("Number of titles: ", counter)

output will be like this one:

...
Title: Maybe we shouldn't let grandma decide things anymore.  
Comments: 2018
---- 
Title: Carrie Fisher and Her Stunt Double Sunbathing on the Set of Return of The Jedi, 1982  
Comments: 880
---- 
Title: fidget spinner  
Comments: 1537
---- 
Number of titles:  25

score 7 · Answer 3 · answered Jan 31 '16 at 22:24

7

I would usually use the requests package with the json package. The following code should be suitable for your needs:

import requests
import json

url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
r = requests.get(url)
print(json.loads(r.content))

Output

answered Jan 31 '16 at 22:24

gtlambert

11,711
2
30
48

Using requests is really the most easiest to use solution to this. – ferdy Jan 31 '16 at 22:28
1

I'm using Python 3.5 and am getting an error: AttributeError: module 'requests' has no attribute 'get' Any ideas on how to resolve this? – ClickThisNick Jan 31 '16 at 22:35
1

You might not have it installed - try running `pip install requests` from the command line – gtlambert Jan 31 '16 at 22:38
1

I have tried this, but I get the error: `the JSON object must be str, not 'bytes'`... any idea why? – pookie Feb 07 '17 at 15:21
@pookie try using : print(json.loads(r.text)) and you can check out the differences just use : type(r.text) type(r.content) – Lokendra Singh Rawat Dec 29 '17 at 22:50

score 3 · Answer 4 · 2019-10-25T06:47:56.017

The only thing missing in the original question is a call to the decode method on the response object (and even then, not for every python3 version). It's a shame no one pointed that out and everyone jumped on a third party library.

Using only the standard library, for the simplest of use cases :

import json
from urllib.request import urlopen


def get(url, object_hook=None):
    with urlopen(url) as resource:  # 'with' is important to close the resource after use
        return json.load(resource, object_hook=object_hook)

Simple use case :

data = get('http://url') # '{ "id": 1, "$key": 13213654 }'
print(data['id']) # 1
print(data['$key']) # 13213654

Or if you prefer, but riskier :

from types import SimpleNamespace

data = get('http://url', lambda o: SimpleNamespace(**o)) # '{ "id": 1, "$key": 13213654 }'
print(data.id) # 1
print(data.$key) # invalid syntax
# though you can still do
print(data.__dict__['$key'])

score 1 · Answer 5 · edited Jan 01 '19 at 08:47

1

With Python 3

import requests
import json

url = 'http://IP-Address:8088/ws/v1/cluster/scheduler'
r = requests.get(url)
data = json.loads(r.content.decode())

edited Jan 01 '19 at 08:47

Morse

695
8
8

answered Apr 20 '18 at 00:17

kartik

2,097
3
21
31

`import requests` – Mitchell Currie Sep 06 '18 at 23:53
and `import json` the snippet should work as copy and paste. nice concise answer otherwise – Mitchell Currie Sep 06 '18 at 23:54

Python 3 Get and parse JSON API

5 Answers5

Linked