Error while using threads to scrape a website

Question

I am trying to build a bot that will scrap the purchase history of purchased domains. So far I was able to extract the domain from the csv file and store them into a list (PS: there are 10k domains). The problem accures when I am trying to scrap the website with them. I have tried doing this with two domains and it works perfectly. Does anyone know what error is this and how I can fix it? Thank you very much in advance.

my code:

datafile = open('/Users/.../Documents/Domains.csv', 'r')
myreader = csv.reader(datafile, delimiter=";",)
domains   = []
for row in myreader:
    domains.append(row[1])
del domains[0]
print("The Domains have been stored into a list")

nmb_sells_record = 0

def result_catcher(domains,queue):
    template_url = "https://namebio.com/{}".format(domain)
    get = requests.get(template_url)
    results = get.text
    last_sold =  results[results.index("last sold for ")+15:results.index(" on 2")].replace(",","")
    last_sold = int(last_sold)
    if not "No historical sales found." in results:
        sold_history = results[results.index("<span class=\"label label-success\">"):results.index(" USD</span> on <span class=\"label")]
    queue.put(results)

#domains = ["chosen.com","koalas.com"]
queues = {}
nmb=0
for x in range(len(domains)):
    new_queue = "queue{}".format(nmb)
    queues[new_queue] = queue.Queue()
    nmb += 1
count = 0
for domain in domains:
    for queue in queues: 
        t = threading.Thread(target=result_catcher, args=(domain,queues[queue]))
        t.start()
print("The Requests were all sent, now they are beeing analysed")   
for queue in queues:
    response_domain = queues[queue].get()
    nmb_sells_record = response_domain.count("for $") + response_domain.count("USD")


print("The Bot has recorded {} domain sells".format(nmb_sells_record))

The output of my code:

Exception in thread Thread-345:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x115a55a20>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

Answer 1

From the python docs:

exception socket.gaierror A subclass of OSError, this exception is raised for address-related errors by getaddrinfo() and getnameinfo().

The accompanying value is a pair (error, string) representing an error returned by a library call. string represents the description of error, as returned by the gai_strerror() C function. The numeric error value will match one of the EAI_* constants defined in this module.

gai => get address info

From the urllib3 wikipage:

New exception: NewConnectionError, raised when we fail to establish a new connection, usually ECONNREFUSED socket error.

Some possible reasons for an ECONNREFUSED error here along with some commandline commands to probe the address and port.

By the way, instead of reading all the rows into an array, then deleting the first item in the array, which makes python slide all the other items over one spot, you can more efficiently skip the header(?) like this:

myreader = csv.reader(datafile, delimiter=";",)
next(my_reader)  #<==== HERE ****

domains   = []

for row in myreader:
    domains.append(row[1])

next() will throw a StopIteration exception if there isn't a next row. If you want to prevent that, you can call next(my_reader, None), which will return None if there is no next row.

Threading example:

import requests
import threading

resources = [
    "dfactory.com",
    "dog.com",
    "cat.com",
]

def result_catcher(resource):
    template_url = "https://namebio.com/{}".format(resource)
    get = requests.get(template_url)


threads = []

for resource in resources:
    t = threading.Thread(target=result_catcher, args=(resource,) )
    t.start()
    threads.append(t)

for thread in threads:
    thread.join()

print("All threads done executing.")

By the way, there will be an optimal number of threads to start, which is less than N. Create a thread pool, and when one thread is done have it go back and read another resource path from a worker queue. You'll have to run some tests to figure out how many threads are optimal. Creating 10,000 threads is not optimal. If you have four cores, as few as 10 threads may be optimal.