What is the most efficient way to verify multiple email hostnames in a pandas dataframe

Question

I have a pandas data frame, which has a column of the hostname of each email address (over 1000 rows):

email               hostname
email@example.com   example.com
email2@example.com  example.com
email@example2.com  example2.com
email@example3.com  example3.com

I want to go through each hostname and check whether it truly exists or not.

email               hostname      valid_hostname
email@example.com   example.com   True
email2@example.com  example.com   False
email@example2.com  example2.com  False
email@example3.com  example3.com  False

First, I extracted the hostname of each email address:

df['hostname'] = df['email'].str.split('@').str[1]

Then, I tried to check the DNS using pyIsEmail, but that was too slow:

from pyisemail import is_email    
df['valid_hostname'] = df['hostname'].apply(lambda x: is_email(x, check_dns=True))

Then, I tried a multi-threaded function:

import requests
from requests.exceptions import ConnectionError

def validate_hostname_existence(hostname:str):
    try:
        response = requests.get(f'http://{hostname}', timeout=0.5)
    except ConnectionError:
        return False
    else:
        return True

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as executor:
        df['valid_hostname'] = pd.Series(executor.map(validate_hostname_existence, df['hostname']),index=df['hostname'].index)

But that did not go so well, too, as I'm pretty new to parallel functions. It has multiple errors, and I believe it can be much more beneficial if I could somehow first check whether this hostname got checked already and skip the entire request all over again. I would like to go as far as I can without actually sending an email.

Is there a library or a way to accomplish this? As I could not find a proper solution to this problem so far.

Answer 1

You answered your self, You can use cache to save in the memory the hostnames you already checked.

for example:

   from functools import lru_cache
   @lru_cache(max_size=None) 
   def my_is_email(x, check_dns=True):
       return is_email(x, check_dns=check_dns)

It's also recommended to limit the size to prevent memory overflow. for example:

@lru_cache(max_size=256) 

for more information read This

Answer 2

I don't know anything about panda but here's how you can process a list of emails in parallel and get back of a set of valid emails. I'm sure you can adapt this to your panda case.

from queue import Empty, Queue
from threading import Thread, Lock
from pyisemail import is_email

q = Queue()
lock = Lock()
valid = set()

emails = ["test@google.com", "test2@gmail.com"]
for e in emails:
    q.put(e)


def process_queue(queue: Queue):
    while True:
        try:
            email = queue.get(block=False)
        except Empty:
            break
        if is_email(email, check_dns=True):
            lock.acquire()
            valid.add(email)
            lock.release()


NUM_THREADS = 30
threads = []

for i in range(NUM_THREADS):
    thread = Thread(target=process_queue, args=(q,))
    thread.start()
    threads.append(thread)

for thread in threads:
    thread.join()

print("done")
print(valid)

Explanation

1. Create a queue object filled with emails
2. Create NUM_THREADS threads.
3. Each thread pulls from queue. If they get an email it process the email. locks the lock protecting the results set. adds to the set. releases. If there are no emails left the thread terminates.
4. Wait for all threads to terminate.