4

I use pywhois to determine if a domain name is registered or not. Here is my source code. (all permutations from a.net to zzz.net)

#!/usr/bin/env python
import whois  #pip install python-whois
import string
import itertools

def main():
    characters = list(string.ascii_lowercase)
    ##domain names generator
    for r in range(1, 4) :
        for name in itertools.permutations(characters, r) : #from 'a.net' to 'zzz.net'
            url = ''.join(name) + '.net'

            #check if a domain name is registered or not
            try :
                w = whois.whois(url)
            except (whois.parser.PywhoisError):  #NOT FOUND
                print(url)   #unregistered domain names?

if __name__ == '__main__':
    main()

I got the following results:

jv.net
uli.net
vno.net
xni.net

However, all above domain names have already been registered. It is not accurate. Can anyone explain it? There are a lot of errors:

fgets: Connection reset by peer
connect: No route to host
connect: Network is unreachable
connect: Connection refused
Timeout.

There is an alternative way, reported here.

import socket   
try:    
    socket.gethostbyname_ex(url) 
except:
    print(url) #unregistered domain names?

In speaking of speed, I use map to parallel processing.

def select_unregisteredd_domain_names(self, domain_names):
    #Parallelism using map
    pool = ThreadPool(16)  # Sets the pool size
    results = pool.map(query_method(), domain_names)
    pool.close()  #close the pool and wait for the work to finish
    pool.join()

    return results
Community
  • 1
  • 1
SparkAndShine
  • 17,001
  • 22
  • 90
  • 134
  • 1
    Unless I'm missing something, your code just catches errors. That doesn't necessarily mean no registration was found; it could be a network glitch or any number of other issues. – elixenide Jul 23 '15 at 20:14
  • @EdCottrell How to correct my codes? – SparkAndShine Jul 23 '15 at 20:23
  • 1
    I'm not sure what to tell you, as I'm not very familiar with phwhois. That's why I posted a comment instead of an answer. You need to examine the error more closely to determine reasons for the failure. Good luck! – elixenide Jul 23 '15 at 20:30
  • 1
    Querying Verisign for ~40k domains using whois will get you blocked pretty quickly. See my answer below. – Krumelur Jul 23 '15 at 20:43
  • @Krumelur, To speed up, I use **map** to parallel processing in the real code. Something like `results = pool.map(self.whois_domain_name, domain_names)`. – SparkAndShine Jul 23 '15 at 21:10
  • 2
    @SparkandShine I meant you will get blocked by the server. Parallellize and you might get an abuse letter, then you are basically (D)DOS:ing them. – Krumelur Jul 23 '15 at 21:15
  • @Krumelur yeah, a lot of `Timeout` and `Network Unreachable` while running. – SparkAndShine Jul 23 '15 at 21:18

1 Answers1

23

This is a tricky problem to solve, trickier than most people realize. The reason for that is that some people don't want you to find that out. Most domain registrars apply lots of black magic (i.e. lots of TLD-specific hacks) to get the nice listings they provide, and often they get it wrong. Of course, in the end they will know for sure, since they have EPP access that will hold the authoritative answer (but it's usually done only when you click "order").

Your first method (whois) used to be a good one, and I did this on a large scale back in the 90s when everything was more open. Nowadays, many TLDs protect this information behind captchas and obstructive web interfaces, and whatnot. If nothing else, there will be quotas on the number of queries per IP. (And it may be for good reason too, I used to get ridiculous amounts of spam to email addresses used for registering domains). Also note that spamming their WHOIS databases with queries is usually in breach of their terms of use and you might get rate limited, blocked, or even get an abuse report to your ISP.

Your second method (DNS) is usually a lot quicker (but don't use gethostbyname, use Twisted or some other async DNS for efficiency). You need to figure out how the response for taken and free domains look like for each TLD. Just because a domain doesn't resolve doesn't mean its free (it could just be unused). And conversely, some TLDs have landing pages for all nonexisting domains. In some cases it will be impossible to determine using DNS alone.

So, how do you solve it? Not with ease, I'm afraid. For each TLD, you need to figure out how to make clever use of DNS and whois databases, starting with DNS and resorting to other means in the tricky cases. Make sure not to flood whois databases with queries.

Another option is to get API access to one of the registrars, they might offer programmatic access to domain search.

Krumelur
  • 31,081
  • 7
  • 77
  • 119
  • @SparkAndShine So how did you finally manage to resolve this? – meetpd Sep 14 '17 at 10:48
  • @meetpd, I didn't find a better solution so far:-( – SparkAndShine Sep 15 '17 at 00:51
  • If you use an API key from a registrar, you should worry (depending on the trustworthiness of the registrar) about [_domain-name front-running_](https://en.wikipedia.org/wiki/Domain_name_front_running). – jub0bs Jul 16 '19 at 09:03