python requests/urllib3 connection pooling not catching HTTP errors

Question

python requests (urllib3) with connection pooling is not catching http errors. Is this a bug? Or am I doing something wrong?

#!/usr/bin/env python

import contextlib
import requests
import sys

connection_pool_size = 2
adapter = requests.adapters.HTTPAdapter(pool_connections=connection_pool_size,
                                        pool_maxsize=connection_pool_size)
r_session = requests.Session()
r_session.mount('http', adapter)

try:
    with contextlib.closing(r_session.get(sys.argv[1], timeout=30, allow_redirects=True)) as r:
        print 'success %r' % r
except requests.exceptions.HTTPError as e:
    print 'HTTPError %r' % e
except Exception as e:
    print 'Exception %r' % e

output:

$ ./test.py https://github.com
success <Response [200]>
$ ./test.py https://github.com/sithlordyoyoma
success <Response [404]>

I was expecting HTTPError . Am I doing something wrong?

Closing with contextlib I got from this thread should I call close() after urllib.urlopen()?. As suggested by Alex Martelli.

actually running requests without connection also showing this behaviour

#!/usr/bin/env python

import contextlib
import requests
import sys


try:
    with contextlib.closing(requests.get(sys.argv[1], timeout=30, allow_redirects=True)) as r:
        print 'success %r' % r
except requests.exceptions.HTTPError as e:
    print 'HTTPError %r' % e
except Exception as e:
    print 'Exception %r' % e

output:

$ ./test.py https://github.com
success <Response [200]>
$ ./test.py https://github.com/sithlordyoyoma
success <Response [404]>

urllib2 does this correctly

#!/usr/bin/env python

import contextlib
import urllib2
import sys


try:
    with contextlib.closing(urllib2.urlopen(sys.argv[1], timeout=30)) as r:
        print 'success %r' % r
except urllib2.HTTPError as e:
    print 'HTTPError %r' % e
except Exception as e:
    print 'Exception %r' % e

output:

$ ./test.py https://github.com
success <addinfourl at 4338734792 whose fp = <socket._fileobject object at 0x1025a5c50>>
$ ./test.py https://github.com/sithlordyoyoma
HTTPError HTTPError()

`Response.raise_for_status()` may help? Not an answer because I don't understand myself when you're *supposed* to have to use that. — zwol, Nov 12 '15 at 22:38
Isn't 404 an HTTPError? Also I am demonstrating with a 404. But this happens with 400, 502 and 503. The ones I have noticed so far. — krafts, Nov 12 '15 at 22:52
I just updated some changes and few more examples, I was demostrating with a POST instead of a GET for that URL. Also I thought this was only through connection pooling. I can also see this happens when using regular requests. — krafts, Nov 12 '15 at 22:54

score 1 · Accepted Answer · answered Nov 12 '15 at 22:52

1

Regardless of connection pooling, requests.post (and other HTTP methods) does not raise HTTPError on a 404. HTTPError is raised by calling .raise_for_status(), like this example demonstrates:

#!/usr/bin/env python

import requests

r = requests.post(
    'https://github.com/sithlordyoyoma',
    timeout=30,
    allow_redirects=True
)
print 'success %r' % r
r.raise_for_status()

answered Nov 12 '15 at 22:52

Robᵩ

163,533
20
239
308

sigh, read the docs, correct. i was expecting behavior similar to urllib2. And hence the source of the confusion. http://docs.python-requests.org/en/latest/api/#requests.Response.raise_for_status – krafts Nov 12 '15 at 23:02

python requests/urllib3 connection pooling not catching HTTP errors

1 Answers1