Code outputting only one row, rather than table

Question

I've been attempting to scrape a table with multiple rows and columns. Below is the code I've used, the first time I ran it, the results were as expected, however since it just returns one single row of data, the columns are as expected. I can't see what's changed from the first time I've run it, however my python is very basic so I may be missing something obvious.

page_request = requests.get(url)    
soup = BeautifulSoup(page_request.content, 'html.parser')
table = soup.find_all('table')[0]
rows = table.find_all('tr')

for row in rows:
    cols = row.find_all('td')
    cols = [x.text.strip() for x in cols]

I'm sure its something simple, but any help would be appreciated.

Thanks

the value of `cols` is being overwritten each time through the loop, with the data corresponding to each row - so anything you do with it afterwards will only have the data from the last row — Robin Zigmond, Feb 01 '19 at 16:18

score 0 · Accepted Answer · answered Feb 01 '19 at 16:21

Try something like that:

page_request = requests.get(url)    
soup = BeautifulSoup(page_request.content, 'html.parser')
table = soup.find_all('table')[0]
rows = table.find_all('tr')

data = [[td.text.strip() for td in row.find_all('td')] for for row in rows]

Code outputting only one row, rather than table

1 Answers1