Python GET Requests causing 100% CPU Usage Spike

Question

I'm using the python requests library (version 2.4.1) for performing a simple get request, code is below, nothing fancy here. On most website's there are no issues. But on some websites, one in particular www.pricegrabber.com, I experience 100% CPU usage and the code never moves past the point of the get request. No timeout occurs, nothing, just a huge CPU usage spike that never stops.

import requests
url = 'http://www.pricegrabber.com'
r = requests.get(url, timeout=(1, 1))
print 'SUCESS'
print r

score 4 · Accepted Answer · edited May 23 '17 at 10:27

4

Using python 2.7, the latest stable version of the 'requests' library, and enabling logging as shown in this answer indicates that the HTTP request is stuck in a redirect loop.

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.pricegrabber.com
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 301 20
DEBUG:requests.packages.urllib3.connectionpool:"GET /index.php/ut=43bb2597a77557f5 HTTP/1.1" 301 20
DEBUG:requests.packages.urllib3.connectionpool:"GET /?ut=43bb2597a77557f5 HTTP/1.1" 301 20
DEBUG:requests.packages.urllib3.connectionpool:"GET /?ut=43bb2597a77557f5 HTTP/1.1" 301 20
DEBUG:requests.packages.urllib3.connectionpool:"GET /?ut=43bb2597a77557f5 HTTP/1.1" 301 20

...

This continues a bit until:

requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

And the code I used to discover this:

#!/usr/bin/env python

import logging
import requests

logging.basicConfig(level=logging.DEBUG)

url = 'http://www.pricegrabber.com'
r = requests.get(url, timeout=(1, 1))

print 'SUCCESS'
print r

edited May 23 '17 at 10:27

Community

1
1

answered Dec 04 '14 at 17:24

A. R. Younce

1,913
17
22

I don't get that exception. I only get 3 redirects and it gets stuck on DEBUG:requests.packages.urllib3.connectionpool:"GET /?ut=324aba6b1a542668 HTTP/1.1" 301 20 – digitaldavenyc Dec 04 '14 at 17:54
Your experience may vary based on what the website returns to you. Can you find out what version of 'requests' you're using? – A. R. Younce Dec 04 '14 at 17:55
requests 2.4.1. I'm going to upgrade maybe there was a patch for this issue. – digitaldavenyc Dec 04 '14 at 17:57
That fixed it! It looks like there was an issue with version 2.4.1 I updated to 2.5.0 and I get a toomanyredirects – digitaldavenyc Dec 04 '14 at 18:02
@digitaldavenyc yeah, that was a problem in 2.4.1. 2.4.2 had a release bug (problem with the release, not the code) but it was fixed properly in 2.4.3 – Ian Stapleton Cordasco Dec 05 '14 at 14:49

Python GET Requests causing 100% CPU Usage Spike

1 Answers1