Try/except when using Python requests module

Question

Doing some API testing and trying to create a function that given an inputted URL it will return the json response, however if a HTTP error is the response an error message will be returned.

I was using urllib2 before, but now trying to use requests instead. However it looks like my except block is never executed, regardless of the error.

testURL = 'http://httpbin.org/status/404'


def return_json(URL):
    try:
        response = requests.get(URL)
        json_obj = response.json()
        return json_obj
    except requests.exceptions.HTTPError as e:
        return "Error: " + str(e)

The result I get from running the above...

<Response [404]>

Answer 1

If you want the response to raise an exception for a non-200 status code use response.raise_for_status(). Your code would then look like:

testURL = 'http://httpbin.org/status/404'


def return_json(URL):
    response = requests.get(testURL)

    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as e:
        # Whoops it wasn't a 200
        return "Error: " + str(e)

    # Must have been a 200 status code
    json_obj = response.json()
    return json_obj

You can tell that this is clearly simpler than the other solutions here and doesn't require you to check the status code manually. You would also just catch an HTTPError since that is what raise_for_status will raise. Catching RequestsException is a poor idea. That will catch things like ConnectionErrors or TimeoutErrors, etc. None of those mean the same thing as what you're trying to catch.

Answer 2

Note: You should rather go with response.raise_for_status() as described in Ian's answer above (he's one of the maintainers of the requests module).

---

How you handle this all depends on what you consider an HTTP error. There's status codes, but not everything other than 200 necessarily means there's an error of some sort.

As you noticed, the request library considers those just another aspect of a HTTP response and doesn't raise an exception. HTTP status 302 for example means Found, but the response doesn't contain a response body but a Location header instead that you'd need to follow to get to the resource you actually wanted.

So you'll want to look at response.status_code, and do your handling of that, while catching actual protocol errors with a try..except. When catching those you should actually catch requests.exceptions.RequestException, because this is the base class for all other exceptions the requests module raises.

So here's an example that demonstrates all three cases:

• Sucessfull 200 OK response
• Sucessful request and response, but status other than 200
• Protocol error (invalid schema)

import requests

test_urls = ['http://httpbin.org/user-agent',
             'http://httpbin.org/status/404',
             'http://httpbin.org/status/500',
             'httpx://invalid/url']


def return_json(url):
    try:
        response = requests.get(url)

        # Consider any status other than 2xx an error
        if not response.status_code // 100 == 2:
            return "Error: Unexpected response {}".format(response)

        json_obj = response.json()
        return json_obj
    except requests.exceptions.RequestException as e:
        # A serious problem happened, like an SSLError or InvalidURL
        return "Error: {}".format(e)


for url in test_urls:
    print "Fetching URL '{}'".format(url)
    print return_json(url)
    print

Output:

Fetching URL 'http://httpbin.org/user-agent'
{u'user-agent': u'python-requests/2.1.0 CPython/2.7.1 Darwin/11.4.2'}

Fetching URL 'http://httpbin.org/status/404'
Error: Unexpected response <Response [404]>

Fetching URL 'http://httpbin.org/status/500'
Error: Unexpected response <Response [500]>

Fetching URL 'httpx://invalid/url'
Error: No connection adapters were found for 'httpx://invalid/url'

There could also be an exception raised by response.json() if you get a sucessfull response, but it simply isn't JSON - so you might want to account for that as well.

---

Note: The if not response.status_code // 100 == 2 bit works like this: The // operator does a so called floor division, so it rounds down to the next integer (this is the default behavior for the / in Python 2.x, but not Python 3.x, which changed / to do floating point division). So status // 100 == 2 holds true for all 2xx codes.

Answer 3

You can check the response.status_code value. If it's not 200, then you can consider it to be an error condition and throw your own exception.