Given a big list of urls, what is a way to check which are active/inactive?

Question

Suppose I was giving this list of urls:

website.com/thispage

website.com/thatpage

website.com/thispageagain

website.com/thatpageagain

website.com/morepages

... could possibly be over say 1k urls.

What is the best/easiest way to kinda loop through this list and check whether or not the page is up?

possible duplicate of [Python verify url goes to a page](http://stackoverflow.com/questions/4041443/python-verify-url-goes-to-a-page) — finnw, Nov 17 '12 at 10:16

Marcin · Accepted Answer · 2012-11-16T21:49:18.367

6

Perform a HEAD request on each of them.

Use this library: http://docs.python-requests.org/en/latest/user/quickstart/#make-a-request

requests.head('http://httpbin.org/get').status_code

edited Nov 16 '12 at 21:49

answered Nov 16 '12 at 21:38

Marcin

48,559
18
128
201

can provide me a snippet? i remember using something a while back with python but cant think of the name of the module. – iCodeLikeImDrunk Nov 16 '12 at 21:39
@yaojiang Why don't you look at this: http://docs.python-requests.org/en/latest/ and come back when you hit a snag with your own code. – Marcin Nov 16 '12 at 21:40
wonderful, ill check that out! i was thinking something like "url.open(site).getcode" but im sure the syntax and function names are all wrong. – iCodeLikeImDrunk Nov 16 '12 at 21:42
@yaojiang It's about that easy with `requests` – Marcin Nov 16 '12 at 21:46

score 5 · Answer 2 · answered Nov 16 '12 at 22:41

Here is an example in Python

import httplib2

h = httplib2.Http()
listUrls = ['http://www.google.com','http://www.xkcd.com','http://somebadurl.com']
count = 0

for each in listUrls:
    try:
        response, content = h.request(listUrls[count])
        if response.status==200:
            print "UP"
    except httplib2.ServerNotFoundError:
        print "DOWN"
    count = count + 1

score 2 · Answer 3 · edited May 23 '17 at 11:58

2

There's an SO answer showing how to perform HEAD requests in Python:

How do you send a HEAD HTTP request in Python 2?

edited May 23 '17 at 11:58

Community

1
1

answered Nov 16 '12 at 21:42

John Gaines Jr.

11,174
1
25
25

score 1 · Answer 4 · answered Nov 16 '12 at 21:39

1

Open a pool of threads, open a Url for each, wait for a 200 or a 404. Rinse and repeat.

answered Nov 16 '12 at 21:39

enjoylife

3,801
1
24
33

Given a big list of urls, what is a way to check which are active/inactive?

4 Answers4

Linked

Related