python webscraping using multiprocessing and list comprehension very slow

Question

I try to scrape a website using bs4. but the code is very very slow because there are many tr to get one by one.

1. I tried loop but take more than 3 minute to scrape 3000 tr (1000 row):

ROWS = []
#soup.select('tr[id^=mix]') is a list of html elements
for tr in soup.select('tr[id^=mix]'): 
    dt = tr.select_one('.h').text 
    H_team = tr.select_one('td.Home').text.strip()
    A_team = tr.select_one('td.Away').text.strip()
    #....

    row = [dt, H_team, A_team, ...]    
    ROWS.append(row)
    print(row)

2. I tried List Comprehension but didn't change the speed (even slower):

def my_funct(tr):
    dt = tr.select_one('.h').text 
    H_team = tr.select_one('td.Home').text.strip()
    A_team = tr.select_one('td.Away').text.strip()
        
    row = [dt, H_team, A_team]    
    return row 

ROWS = [my_funct(tr) for tr in soup.select('tr[id^=mix]')]

3. I tried multiprocessing module, but the speed is the same

from multiprocessing.dummy import Pool as ThreadPool

def my_funct(tr):
    dt = tr.select_one('.h').text 
    H_team = tr.select_one('td.Home').text.strip()
    A_team = tr.select_one('td.Away').text.strip()
        
    row = [dt, H_team, A_team]    
    return row 

pool = ThreadPool(4)
ROWS = pool.map(my_funct, soup.select('tr[id^=mix]'))

pool.close()
pool.join()

4. I tried asyncio, but didn't work (return error)

import asyncio

async def my_funct(tr):
    dt = tr.select_one('.h').text 
    H_team = tr.select_one('td.Home').text.strip()
    A_team = tr.select_one('td.Away').text.strip()
        
    row = [dt, H_team, A_team]    
    return row 

async def s():
    await asyncio.gather(*[my_funct(tr) for tr in soup.select('tr[id^=Today]')]) 

asyncio.run(s())

#return error: "RuntimeError: asyncio.run() cannot be called from a running event loop"

How can I run the scraping of rows in parallel so my code doesn't take a long time to process each row one by one ?

Answer 1

When it comes to performance there are generally two major reasons for why there might be bottlenecks which are compute or I/O.

I'll assume that web pages are fully loaded during the scraping process which would eliminate network I/O being an issue. If that is not true and the webpages being scraped are paginated, it would be best to first cache all of these pages in memory to improve performance when processing.

It looks like you have tried to perform multiprocessing using threads. Threads use the same memory of the process they belong to which is good since it reduces inter process communication overhead. However due to python's global interpreter lock this will not improve performance in cpu bound python applications as the workload is bottlenecked by a single thread running at a time.

The fact that it performs slower on your dataset is expected as there is now a slight overhead in managing and context switching between threads.

Try switching:

pool = ThreadPool(4)

to

pool = Pool(4) # number of processors available

Benchmarking a smaller dataset with various number of processors may help identify the optimal amount.

Answer 2

As you do not share your data, I can only guess: you try to parse a big table with many N columns and M=3000 rows.

Your current implementation calls select_one("td...") M x N times. This may be what makes your code slow.

What you can try: Get each row with one select.

ls = []
for tr in soup.select('tr[id^=mix]'): 
    row = [td.get_text(strip=True) for td in tr.select('td')]
    ls.append(row)
df = pd.Dataframe(ls, columns=["...in the right order.."])

Here is a benchmark

def generate_html_table(n_row, n_col):
    tds = [f'<td class="c{i}">{i}</td>' for i in range(n_col)]
    tr = "<tr>" + "".join(tds) + "</tr>"
    table = "<table>" + tr * n_row + "</table>"
    return table

## Generate a html table of 1000 x 20, with class attribute for each column
M, N = 1000, 20
soup = BeautifulSoup(generate_html_table(M, N))

columns = [f"col_{i}" for i in range(N)]

# OP approach: one .select_one for each td
def parse_table_1(soup):
    rows = []
    for tr in soup.select('tr'): 
        row = [tr.select_one(f'td.c{i}').get_text(strip=True) for i in range(N)]
        rows.append(row)
    return pd.DataFrame(rows, columns=columns)

# Proposed approach: one .select for each row
def parse_table_2(soup):
    rows = []
    for tr in soup.select('tr'): 
        row = [td.get_text(strip=True) for td in tr.select("td")]
        rows.append(row)
    return pd.DataFrame(rows, columns=columns)

Results: 10X speed up for a 1000 x 20 table

%timeit parse_table_1(soup)
3.69 s ± 328 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit parse_table_2(soup)
351 ms ± 39.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

For 1000 x 100 table, it's 35x speed up

1min 9s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)
1.88 s ± 175 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)