Doing a whois with a list of domain names

Question

I've got a file of domain names e.g. equivalent to 2500.

I would like to do a whois on these domain names.

Issue is I've never done that and don't know where to start. If you have any ideas, I'm all ears.

TIA.

Answer 1

You can also use the Linux commandtool whois. The following code opens a subprocess and searches for the domain.

But you have to be carefull with many requests in short time. The servers will eventually block you after a while. ;)

import subprocess

def find_whois(domain):
    # Linux 'whois' command wrapper
    # 
    # Executes a whois lookup with the linux command whois.
    # Returncodes from: https://github.com/rfc1036/whois/blob/master/whois.c

    domain = domain.lower().strip()
    d = domain.split('.')
    if d[0] == 'www': d = d[1:]

    # Run command with timeout
    proc = subprocess.Popen(['whois', domain], stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    ans,err = proc.communicate(input)

    if err == 1: raise WhoisError('No Whois Server for this TLD or wrong query syntax') 
    elif err == 2: raise WhoisError('Whois has timed out after ' + str(whois_timeout) + ' seconds. (try again later or try higher timeout)')
    ans = ans.decode('UTF-8')
    return ans


with open('domains.txt') as input:
    with open('out.txt','a') as output:
        for line in input:
            output.write(find_whois(line))

The with open as statement handles the filestream. The 'a' at the output file means the file is opened in append-mode.

Answer 2

It looks like you've had some helpful answers already, but I thought it might be good to say a little more about the challenges of doing WHOIS lookups in bulk (and in general) and provide some alternative solutions.

The WHOIS lookup

Looking up a single domain name typically involves finding the relevant WHOIS server for that domain and then requesting the information via port 43. If you have access to a unix-like shell (e.g. Bash), you can use whois to do this easily (as noted by others):

$ whois example.com

Very similar WHOIS tools have also been made available as modules for a vast array of programming languages. The pywhois module for Python is one example.

In its simplest form, a bulk WHOIS lookup is just looping over a list of domains, issuing a whois request for each domain and writing the record to an output.

Here is an example in Bash that reads domains from a file domains.txt and writes each WHOIS record into separate files (if you're using Windows, give Cygwin a try).

#!/bin/bash

domain_list="domains.txt"

while read line 
do
    name=$line
    echo "Looking up ${line}..."
    whois $name > ${line}.txt
    sleep 1
done < $domain_list

Beware of the following complications of WHOIS lookups in bulk:

• Some WHOIS servers may not give you a full WHOIS record. This is especially true for country-specific domains (such as .de and .fr) and domains registered with certain registrars (such as GoDaddy).

  If you want the fullest possible record, you'll often have to go to the registry's website or to a third-party service which may have cached the record (e.g. DomainTools). This is much more difficult to automate and may have to be done manually. Even then, the record may not contain what you want (e.g. contact details for the registrant).

• Some WHOIS servers impose restrictions on the number of requests you can make in a certain time frame. If you hit the limit, you might find that you have to return a few hours later to request the records again. For example, with .org domains, you limited to no more than three lookups in a minute and a few registrars will bar you for 24 hours.

  It's best to pause for a few seconds between lookups, or try to shuffle your list of domains by TLD so you don't bother the same server too many times in quick succession.

• Some WHOIS servers are frequently down and the request will time out, meaning that you might need to go back and re-do these lookups. ICANN stipulates that whois servers must have a pretty decent uptime, but I've found one or two servers that are terrible at giving out records.

Parsing the record

Parsing WHOIS records (e.g. for registrant contact information) can be a challenge because:

• The records are not always in a consistent format. You'll find this with the .com domains in particular. A .com record might be held any one of thousands of registrars worldwide (not by the .com registry, Verisign) and not all choose to present the records in an easy-to-parse format recommended by ICANN.

• Again, the information you want to extract might not be in the record you get back from the lookup.

Since it's been mentioned already, pywhois is one option to parse WHOIS data. Here's a very simple Python script which looks up the WHOIS record for each domain and extracts the registrant name (where possible*), writing the results to a CSV file. You can include other fields too if you like:

import whois
import csv

with open("domains.txt", "r") as f:
    domains = f.readlines()
    
with open("output_file.csv", "wb") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(["Domain", "Registrant Name"])
    for domain in domains:
        domain = domain.rstrip()
        record = whois.whois(domain)
        try:
            r_name = record.registrant_name
        except AttributeError:
            r_name = "error"
        writer.writerow([domain, r_name])

* When I tested this script quickly, pywhois wasn't very reliable in extracting the registrant name. Another similar library you could try instead is pythonwhois.

Answer 3

Assuming the domains are in a file named domains.txt and you have pywhois installed, then something like this should do the trick:

import whois

infile = "domains.txt"

# get domains from file
with open(infile, 'rb') as f:
    domains = [line.rstrip() for line in f if line.rstrip()]

for domain in domains:
    print domain
    record = whois.whois(domain)

    # write each whois record to a file {domain}.txt
    with open("%s.txt" % domain, 'wb') as f:
        f.write(record.text)

This will output each whois record to a file named {domain}.txt

---

Without pywhois:

import subprocess

infile = "domains.txt"

# get domains from file
with open(infile, 'rb') as f:
    domains = [line.rstrip() for line in f if line.rstrip()]

for domain in domains:
    print domain
    record = subprocess.check_output(["whois", domain])

    # write each whois record to a file {domain}.txt
    with open("%s.txt" % domain, 'wb') as f:
        f.write(record)

Answer 4

Download and install Microsoft's whois tool from http://technet.microsoft.com/en-us/sysinternals/bb897435.aspx

Create a text file with the list of domain names, with a header row.

name
google.com
yahoo.com
stackoverflow.com

Create a powershell script:

$domainname = Import-Csv -Path "C:\domains.txt"
foreach($domain in $domainname) 
{
   .\whois.exe $domain.name Export-Csv -Path "C:\domain-info.csv" -Append
}

Run the powershell script.

Answer 5

You can do that with a simple "one liner" with the command xargs.

xargs -n 1 -a valid_dns.txt -I {} sh -c 'echo "Domain: {}"; whois {}'

++