Use httplib to check if a URL will return a certain page?

Question

Im going through several hundred bit.ly links to see if they have been used to shorten a link. If a link has not, it returns this page.

How can I iterate through a list of links to check which ones do NOT return this page?

I tried using the head method used in this question, however that always returned true, of course.

I looked into the head method, but found out that it never returns any data:

>>> import httplib
>>> conn = httplib.HTTPConnection("www.python.org")
>>> conn.request("HEAD","/index.html")
>>> res = conn.getresponse()
>>> print res.status, res.reason
200 OK
>>> data = res.read()
>>> print len(data)
0
>>> data == ''
True

I'm stumped on this, and any help would be great.

I was hoping that it would be possible to check the link without loading the content of the page, but if thats the only way it can be done then so be it — Scherf, Mar 02 '14 at 22:24
@Scherf: I've tried your link from the question and it returns 404 for me (404 is *not* redirect, it is "not found"). — jfs, Mar 02 '14 at 22:42

jfs · Accepted Answer · 2014-03-02T22:53:05.693

If bit.ly returns 404 http code for non-shorten links:

#!/usr/bin/env python
from httplib import HTTPConnection
from urlparse import urlsplit

urls = ["http://bit.ly/NKEIV8", "http://bit.ly/1niCdh9"]
for url in urls:
    host, path = urlsplit(url)[1:3]
    conn = HTTPConnection(host)
    conn.request("HEAD", path)
    r = conn.getresponse()
    if r.status != 404:
       print("{r.status} {url}".format(**vars()))

Unrelated: to speed up the check, you could use multiple threads:

#!/usr/bin/env python
from httplib import HTTPConnection
from multiprocessing.dummy import Pool # use threads
from urlparse import urlsplit

def getstatus(url):
    try:
        host, path = urlsplit(url)[1:3]
        conn = HTTPConnection(host)
        conn.request("HEAD", path)
        r = conn.getresponse()
    except Exception as e:
        return url, None, str(e) # error
    else:
        return url, r.status, None

p = Pool(20) # use 20 concurrent connections
for url, status, error in p.imap_unordered(getstatus, urls):
    if status != 404:
       print("{status} {url} {error}".format(**vars()))

This works perfectly. The issue I was having is that when I imported the links from file I forgot to remove the newline character, and because of that they all returned 200. Great answer, thanks — Scherf, Mar 02 '14 at 22:53
@Scherf: `urls` in the code is already a list of urls (any iterable will do). Do you mean how to read that list from a file: `urls = [line.strip() for line in open('filename') if line.strip()]` — jfs, Mar 02 '14 at 23:00

score 0 · Answer 2 · answered Mar 02 '14 at 22:25

0

So, here is a simple way to do that:

import httplib2
h = httplib2.Http(".cache")
resp, content = h.request("http://www.python.org/", "GET")
print content

Source: https://code.google.com/p/httplib2/wiki/Examples

answered Mar 02 '14 at 22:25

Sharif Mamun

3,508
5
32
51

Use httplib to check if a URL will return a certain page?

2 Answers2