If request timeout skip url python requests

Question

I would like the following script to try every url in url_list, and if it exist print it exist(url) and if not print don't(url) and if request timeout skip to the next url using "requests" lib:

url_list = ['www.google.com','www.urlthatwilltimeout.com','www.urlthatdon\'t exist']

def exist:
    if request.status_code == 200:
    print"exist{0}".format(url)
else:
    print"don\'t{0}".format(url)
a = 0

while (a < 2):
url = urllist[a]
try:
    request = requests.get(url, timeout=10)
except request.timeout:#any option that is similar?
    print"timed out"
    continue
validate()
a+=1

Possible duplicate of [Timeout for python requests.get entire response](http://stackoverflow.com/questions/21965484/timeout-for-python-requests-get-entire-response) — Grainier, Jul 16 '16 at 14:22
@Grainier I would like it to continue fetching through the urls after the request time's out, ??? — Elhamer Yacine, Jul 16 '16 at 14:33

score 2 · Accepted Answer · edited May 23 '17 at 10:34

Based on this SO answer below is code which will limit the total time taken by a GET request as well as discern other exceptions that may happen.

Note that in requests 2.4.0 and later you may specify a connection timeout and read timeout by using the syntax:

requests.get(..., timeout=(...conn timeout..., ...read timeout...))

The read timeout, however, only specifies the timeout between individual read calls, not a timeout for the entire request.

Code:

import requests
import eventlet
eventlet.monkey_patch()

url_list = ['http://localhost:3000/delay/0',
            'http://localhost:3000/delay/20', 
            'http://localhost:3333/',         # no server listening
            'http://www.google.com'
           ]

for url in url_list:
     try:
        with eventlet.timeout.Timeout(1):
          response = requests.get(url)
        print "OK -", url
     except requests.exceptions.ReadTimeout:
        print "READ TIMED OUT -", url
     except requests.exceptions.ConnectionError:
        print "CONNECT ERROR -", url
     except eventlet.timeout.Timeout, e:
        print "TOTAL TIMEOUT -", url
     except requests.exceptions.RequestException, e:
        print "OTHER REQUESTS EXCEPTION -", url, e

And here is an express server you can use to test it:

var express = require('express');
var sleep = require('sleep')
var app = express();

app.get('/delay/:secs', function(req, res) {
  var secs = parseInt( req.params.secs )
  sleep.sleep(secs)
  res.send('Done sleeping for ' + secs + ' seconds')  
});

app.listen(3000, function () {
  console.log('Example app listening on port 3000!');
});

It is a way to install `eventlet` in Anaconda? – redeemefy Oct 01 '16 at 21:25 — redeemefy, Oct 01 '16 at 21:25

If request timeout skip url python requests

1 Answers1