85

Is there any standard way of getting JSON data from RESTful service using Python?

I need to use kerberos for authentication.

some snippet would help.

Bala
  • 4,427
  • 6
  • 26
  • 29
  • This may help you http://stackoverflow.com/questions/713847/recommendations-of-python-rest-web-services-framework – Sreenath Nannat Oct 13 '11 at 07:14
  • 2
    I'm not looking fro " Python-**based** REST frameworks". I want to use RESTful service provided by some java server in python. Thanks anyway. – Bala Oct 13 '11 at 07:18

5 Answers5

127

I would give the requests library a try for this. Essentially just a much easier to use wrapper around the standard library modules (i.e. urllib2, httplib2, etc.) you would use for the same thing. For example, to fetch json data from a url that requires basic authentication would look like this:

import requests

response = requests.get('http://thedataishere.com',
                         auth=('user', 'password'))
data = response.json()

For kerberos authentication the requests project has the reqests-kerberos library which provides a kerberos authentication class that you can use with requests:

import requests
from requests_kerberos import HTTPKerberosAuth

response = requests.get('http://thedataishere.com',
                         auth=HTTPKerberosAuth())
data = response.json()
Mark Gemmill
  • 5,889
  • 2
  • 27
  • 22
  • 5
    If you're missing the `requests` module, simply do: `pip install requests`. More info and docs [here](http://docs.python-requests.org/en/latest/user/install/) – benscabbia Feb 02 '16 at 20:18
  • here why my json response become with u before the key, value pair? *{u'status': u'FINISHED', u'startTime': u'2016-11-08T15:32:33.241Z', u'jobId': u'f9d71eaa-d439-4a39-a258-54220b14f1b8', u'context': u'sql-context', u'duration': u'0.061 secs'}* – KARTHIKEYAN.A Nov 09 '16 at 05:31
78

Something like this should work unless I'm missing the point:

import json
import urllib2
json.load(urllib2.urlopen("url"))
Trufa
  • 39,971
  • 43
  • 126
  • 190
  • this would work if there are no credential required to pass. But I get this "urllib2.HTTPError: HTTP Error 401: Unauthorized" error – Bala Oct 13 '11 at 08:09
  • Where are you trying to download from? – Trufa Oct 13 '11 at 08:12
  • 1
    I need to use Kerberos authentication. Sorry, I forgot to mention in question. – Bala Oct 13 '11 at 08:14
  • @BalamuruganK what OS are you using? – Trufa Oct 13 '11 at 08:35
  • I'm using unix. trying with kerberos lib to get token to pass it to httpConnection.putheader('Authorization', ?) – Bala Oct 13 '11 at 08:42
  • @BalamuruganK I won't really be able to help you until I get to my linux box tomorrow. I'm on a PC and I can't try it out now. Sorry! But if I remember correctly, urllib2 has an addheaders() method that might help you too. I'll check back tomorrow to see how you did. I would still recommend you to edit and improve your question with more information on what you are trying to do. – Trufa Oct 13 '11 at 08:52
  • I am not the downvoter, but urllib2 does not look as nice as the requests library, at a glance. – Prof. Falken Oct 18 '12 at 09:29
  • @AmigableClarkKant I wanted to give another choice other than what [Mark](http://stackoverflow.com/a/7750945/463065) suggested. It seemed to have fitted the OPs need since it's the accepted answer but I agree if might not be the best solution for all cases. – Trufa Oct 18 '12 at 16:26
  • When Python 3 instead of 'urllib2' use 'urllib.request' – csoria Jun 06 '14 at 21:19
27

You basically need to make a HTTP request to the service, and then parse the body of the response. I like to use httplib2 for it:

import httplib2 as http
import json

try:
    from urlparse import urlparse
except ImportError:
    from urllib.parse import urlparse

headers = {
    'Accept': 'application/json',
    'Content-Type': 'application/json; charset=UTF-8'
}

uri = 'http://yourservice.com'
path = '/path/to/resource/'

target = urlparse(uri+path)
method = 'GET'
body = ''

h = http.Http()

# If you need authentication some example:
if auth:
    h.add_credentials(auth.user, auth.password)

response, content = h.request(
        target.geturl(),
        method,
        body,
        headers)

# assume that content is a json reply
# parse content with the json module
data = json.loads(content)
Christo Buschek
  • 296
  • 2
  • 2
10

If you desire to use Python 3, you can use the following:

import json
import urllib.request
req = urllib.request.Request('url')
with urllib.request.urlopen(req) as response:
    result = json.loads(response.readall().decode('utf-8'))
Andre Wisplinghoff
  • 155
  • 1
  • 5
3

Well first of all I think rolling out your own solution for this all you need is urllib2 or httplib2 . Anyways in case you do require a generic REST client check this out .

https://github.com/scastillo/siesta

However i think the feature set of the library will not work for most web services because they shall probably using oauth etc .. . Also I don't like the fact that it is written over httplib which is a pain as compared to httplib2 still should work for you if you don't have to handle a lot of redirections etc ..

dusual
  • 2,097
  • 3
  • 19
  • 26