HTTP requests and JSON parsing in Python

Question

I want to dynamically query Google Maps through the Google Directions API. As an example, this request calculates the route from Chicago, IL to Los Angeles, CA via two waypoints in Joplin, MO and Oklahoma City, OK:

http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false

It returns a result in the JSON format.

How can I do this in Python? I want to send such a request, receive the result and parse it.

See also https://stackoverflow.com/questions/20199126/reading-json-from-a-file, for the case where the JSON data is already in a file, or where you already have code that accesses the API using the standard library `urllib` or successors. (Note that the response object in the latter case is a file-like object.) — Karl Knechtel, Jul 02 '22 at 02:10
[Another great answer can be found here in a different post](https://stackoverflow.com/a/28069789/5675325). — Tiago Martins Peres, Sep 09 '22 at 11:38
How is this a duplicate of a question asked after this one? Also, how is this a duplicate of a question that asks someting totally different? — osiris, Apr 01 '23 at 19:12

score 498 · Accepted Answer · edited Feb 07 '20 at 16:37

498

I recommend using the awesome requests library:

import requests

url = 'http://maps.googleapis.com/maps/api/directions/json'

params = dict(
    origin='Chicago,IL',
    destination='Los+Angeles,CA',
    waypoints='Joplin,MO|Oklahoma+City,OK',
    sensor='false'
)

resp = requests.get(url=url, params=params)
data = resp.json() # Check the JSON Response Content documentation below

JSON Response Content: https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content

edited Feb 07 '20 at 16:37

iinuwa

444
4
14

answered Jun 17 '11 at 13:22

Zach Kelling

52,505
13
109
108

7

For me, I needed to do `json=params` instead of `params=params` or I get a 500 error. – demongolem Nov 06 '18 at 16:07

score 170 · Answer 2 · edited Feb 14 '20 at 05:11

170

The requests Python module takes care of both retrieving JSON data and decoding it, due to its builtin JSON decoder. Here is an example taken from the module's documentation:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...

So there is no use of having to use some separate module for decoding JSON.

edited Feb 14 '20 at 05:11

wesinat0r

121
8

answered Jul 08 '13 at 00:06

linkyndy

17,038
20
114
194

4

If you need to be compatible with requests 0.x (Debian wheezy), you should use `json.load()` or `json.loads()` instead, as in 0.x, `json` is a property rather than a function. – alexia Dec 09 '13 at 14:56
2

@nyuszika If you are using debian, if somehow possible, use pip to get newer python libraries. You don't want to code with old python libraries, unless there is an important reason to use what debian has in the apt repositories. – SHernandez Jun 10 '14 at 12:46
@SHernandez That's a valid point, but some packages might depend on the `python-requests` (or `python3-requests`) package, so you will need to install somewhere else than `/usr/local` to avoid breaking those packages. On the other hand, when portability/compatibility is trivial, in my opinion it's worth it. – alexia Jun 10 '14 at 20:53
Thats true. I try to use a virtual environments for all user space projects and inside those use a local pip – SHernandez Jun 10 '14 at 21:02
3

How to extract only a particular name-value pair from the json response 'r' ? – 3lokh Jan 17 '15 at 16:37
1

In `r.json()` (from my answer) you have the actual response, JSON-decoded. You can access it like a normal `list`/`dict`; `print r.json()` to see how it looks like. Or refer to the API docs of the service you've made the request for. – linkyndy Jan 17 '15 at 18:16
@NikhilGeorge It doesn't appear that .json is chainable. The only way I have been able to extract a particular key-value pair is to assign the results to an intermediate variable. `r = requests.get('https://github.com/timeline.json') myJson = r.json() print (myJson['url'])` – Not a machine Apr 14 '17 at 19:43
this method needs much more memory – akdora Aug 03 '17 at 11:55

score 50 · Answer 3 · edited Aug 22 '17 at 01:13

50

requests has built-in .json() method

import requests
requests.get(url).json()

edited Aug 22 '17 at 01:13

JamesThomasMoon

6,169
7
37
63

answered Nov 05 '13 at 09:19

maow

1,387
11
20

score 32 · Answer 4 · answered Jun 17 '11 at 13:22

32

import urllib
import json

url = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
result = json.load(urllib.urlopen(url))

answered Jun 17 '11 at 13:22

clyfish

10,240
2
31
23

3

Thanks for your help, however the following is to be noted : The urllib.urlopen() function has been removed in Python 3.0 in favor of urllib2.urlopen(). – Arun Jun 17 '11 at 14:00
2

Arun, yes but it's no longer named urllib2 – Corey Goldberg Jun 17 '11 at 15:09
6

It's `urllib.request` in Python 3. – alexia Jun 10 '14 at 20:55
It does not work. json.loads gives 'TypeError: the JSON object must be str, not 'HTTPResponse'' and json.load gives 'TypeError: the JSON object must be str, not 'bytes'' – Menachem Hornbacher Dec 23 '15 at 17:46
wrong again @nyuszika7h. it gonna gave you 'module object is not callable' – greendino May 23 '21 at 21:17
@Arun ERROR: No matching distribution found for urllib2. are you sure? – greendino May 23 '21 at 21:19
@greendino I meant `urllib.request` instead of `urllib(2)`, so `urllib.request.urlopen`. Also make sure to `import urllib.request` and not just `import urllib`. – alexia Jan 17 '22 at 10:25

Michael · Answer 5 · 2013-03-22T05:56:10.293

21

Use the requests library, pretty print the results so you can better locate the keys/values you want to extract, and then use nested for loops to parse the data. In the example I extract step by step driving directions.

import json, requests, pprint

url = 'http://maps.googleapis.com/maps/api/directions/json?'

params = dict(
    origin='Chicago,IL',
    destination='Los+Angeles,CA',
    waypoints='Joplin,MO|Oklahoma+City,OK',
    sensor='false'
)


data = requests.get(url=url, params=params)
binary = data.content
output = json.loads(binary)

# test to see if the request was valid
#print output['status']

# output all of the results
#pprint.pprint(output)

# step-by-step directions
for route in output['routes']:
        for leg in route['legs']:
            for step in leg['steps']:
                print step['html_instructions']

edited Mar 22 '13 at 05:56

answered Mar 22 '13 at 01:23

Michael

1,177
4
20
71

Michael, how can I make some sense out of this once I got the data? How do I display it in "classic" json visual format (like the one you get in your browser)? Here is what I get in my terminal: [link]http://s13.postimg.org/3r55jajk7/terminal.png – Alexander Starbuck Feb 16 '16 at 09:16
3

@AlexStarbuck `import pprint` then -> `pprint.pprint(step['html_instructions'])` – Michael Feb 16 '16 at 14:20

score 13 · Answer 6 · edited Jul 08 '21 at 08:04

13

just import requests and use from json() method :

source = requests.get("url").json()
print(source)

OR you can use this :

import json,urllib.request
data = urllib.request.urlopen("url").read()
output = json.loads(data)
print (output)

edited Jul 08 '21 at 08:04

dejanualex

3,872
6
22
37

answered Jul 16 '20 at 07:26

mamal

1,791
20
14

do i need to use something like data.close() after the code? or the request gets closed automatically after the url has been called. – Angad Cheema Feb 24 '23 at 19:58

score 11 · Answer 7 · answered Mar 31 '18 at 19:06

Try this:

import requests
import json

# Goole Maps API.
link = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'

# Request data from link as 'str'
data = requests.get(link).text

# convert 'str' to Json
data = json.loads(data)

# Now you can access Json 
for i in data['routes'][0]['legs'][0]['steps']:
    lattitude = i['start_location']['lat']
    longitude = i['start_location']['lng']
    print('{}, {}'.format(lattitude, longitude))

score 4 · Answer 8 · edited Mar 24 '23 at 11:12

4

Also for pretty Json on console:

import json     
json.dumps(response.json(), indent=2)

possible to use dumps with indent. (Please import json)

edited Mar 24 '23 at 11:12

Garbit

5,805
6
39
72

answered Apr 18 '20 at 14:15

devugur

1,339
1
19
25

HTTP requests and JSON parsing in Python

8 Answers8

Linked

Related