Precisely catch DNS error with Python requests

Question

I am trying to make a check for expired domain name with python-requests.

import requests

try:
    status = requests.head('http://wowsucherror')
except requests.ConnectionError as exc:
    print(exc)

This code looks too generic. It produces the following output:

HTTPConnectionPool(host='wowsucherror', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))

What I'd like to do is to catch this DNS error only (like ERR_NAME_NOT_RESOLVED in Chrome). As a last resort I can just do string matching, but maybe there is a better, more structured and forward compatible way of dealing with this error?

Ideally it should be some DNSError extension to requests.

UPDATE: The error on Linux is different.

HTTPConnectionPool(host='wowsucherror', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))

Reported bug to requests -> urllib3 https://github.com/shazow/urllib3/issues/1003

UPDATE2: OS X also reports different error.

requests.exceptions.ConnectionError: HTTPConnectionPool(host='wowsucherror', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

I think you will be stuck parsing the errno from the string, the socket error is caught here https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connection.py#L149 but no errno attribute is set anywhere so all you get is the error message. If you actually had access to `e` if you just be a matter of checking the e.errno. — Padraic Cunningham, Oct 20 '16 at 10:03
@PadraicCunningham it also looks like that error message it not cross-platform, and I need to know how it looks on Linux and OS X. — anatoly techtonik, Oct 20 '16 at 10:05
indeed, it throws a [errno -2] on my Ubuntu box, I tried `except (NewConnectionError, socket.error) as exc:` but the socket error gets swallowed up. Might be worth starting an issue as it seems like a reasonable thing to want to do, it would just be a matter of passing e.errno along. — Padraic Cunningham, Oct 20 '16 at 10:34

score 7 · Accepted Answer · edited Dec 19 '21 at 21:17

Done this with this hack, but please monitor https://github.com/psf/requests/issues/3630 for a proper way to appear.

# for Python 2 compatibility
from __future__ import print_function
import requests

def sitecheck(url):
    status = None
    message = ''
    try:
        resp = requests.head('http://' + url)
        status = str(resp.status_code)
    if ("[Errno 11001] getaddrinfo failed" in str(exc) or     # Windows
        "[Errno -2] Name or service not known" in str(exc) or # Linux
        "[Errno 8] nodename nor servname " in str(exc)):      # OS X
        message = 'DNSLookupError'
    else:
        raise

    return url, status, message

print(sitecheck('wowsucherror'))
print(sitecheck('google.com'))

score 1 · Answer 2 · answered Oct 30 '19 at 05:39

You could use lower-level network interface, socket.getaddrinfo https://docs.python.org/3/library/socket.html#socket.getaddrinfo

import socket


def dns_lookup(host):
    try:
        socket.getaddrinfo(host, 80)
    except socket.gaierror:
        return False
    return True


print(dns_lookup('wowsucherror'))
print(dns_lookup('google.com'))

score 1 · Answer 3 · answered Dec 10 '20 at 01:45

I have a function based on the earlier answer above which seems to no longer work. This function checks "liveness" of a url based on resolution and also the requests .ok function rather than just specific to resolution errors but can be adapted easily to suit.

def check_live(url):
    try:
        r = requests.get(url)
        live = r.ok
    except requests.ConnectionError as e:
        if 'MaxRetryError' not in str(e.args) or 'NewConnectionError' not in str(e.args):
            raise
        if "[Errno 8]" in str(e) or "[Errno 11001]" in str(e) or ["Errno -2"] in str(e):
            print('DNSLookupError')
            live = False
        else:
            raise
    except:
        raise
    return live

Precisely catch DNS error with Python requests

3 Answers3

Linked