How to speed up scraping with Selenium (multiprocessing)

Question

I am trying to scrape a singular data point from a list of urls to dynamically loaded sites. I have implemented a scraper with selenium, but it is too slow. I tried using scrapy but realized scrapy does not work with dynamically loaded sites. I have seen documentation on splash with scrapy - but this seems to be the case where splash loads one dynamic site and scrapy parses the data from the one site; I have a huge list of urls. I am considering using mutliprocessing but unsure where to get started/if it would work well with selenium.

def get_cost(url):
driver.get(url)
try:
    element = WebDriverWait(driver, 4).until(
        EC.presence_of_element_located((By.XPATH,'/html/body/c-wiz[2]/div/div[2]/c-wiz/div/c-wiz/c-wiz/div[2]/div[2]/ul[1]/li[1]/div/div[2]/div/div[9]/div[2]/span'))
    )
    cost = element.get_attribute('textContent')
except:
    cost = "-"
finally:
    driver.quit()
return cost

This is a function that given a url, grabs the cheapest flight cost on the site. I am very new to web scraping so I would appreciate some advice with the best way to move forward.

Yes it is. When I ran this code on about 200 URLs is was able to correctly grab the information from the site I needed with the Xpath. — max, Jun 10 '22 at 13:04

sound wave · Accepted Answer · 2022-06-10T14:33:55.367

This script uses threading (instead of multiprocessing) to open multiple independent windows (instances) of the browser. This means that the code contained in the function get_cost is run simultaneously in each window.

Since each thread opens a new window, if you have many urls you should open only a small number of urls at a time, for example 10 urls at a time, otherwise the computer may freeze

from selenium import webdriver
import threading

def get_cost(url, costs):

    driver = ...
    driver.get(url)
    try:
        element = WebDriverWait(driver, 4).until(
            EC.presence_of_element_located((By.XPATH,'/html/body/c-wiz[2]/div/div[2]/c-wiz/div/c-wiz/c-wiz/div[2]/div[2]/ul[1]/li[1]/div/div[2]/div/div[9]/div[2]/span'))
        )
        cost = element.get_attribute('textContent')
    except:
        cost = "-"
    finally:
        driver.quit()
    costs.append(cost)

thread_list = []
costs = []
urls = ['...', '...', '...'] # each one is opened in a separate browser)

for idx, url in enumerate(urls):
    t = threading.Thread(name=f'Thread {idx}', target=get_cost, args=(url, costs))
    t.start()
    print(t.name + ' started')
    thread_list.append(t)

# wait for all threads to complete
for thread in thread_list:
    thread.join()

print(costs)

This is fine although I would worry about OP's statement that the URL list is "huge". Should perhaps consider using a thread pool of some kind to constrain the level of concurrency. ThreadPoolExecutor might be a good idea — DarkKnight, Jun 10 '22 at 17:58

How to speed up scraping with Selenium (multiprocessing)

1 Answers1

Linked