BeautifulSoup find_all limited to 50 results?

Question

I'm trying to get the results from a page using BeautifulSoup:

req_url = 'http://www.xscores.com/soccer/livescores/25-02'
request = requests.get(req_url)
content = request.content
soup = BeautifulSoup(content, "html.parser")
scores = soup.find_all('tr', {'style': 'height:18px;'}, limit=None)
print(len(scores))
>50

I read this previous solution: Beautiful Soup findAll doesn't find them all and I tried html.parser, lxml and html5lib, but none of them return more than 50 results. Any suggestions?

score 2 · Accepted Answer · answered Feb 27 '17 at 09:41

2

Try using css-selector query.

scores = soup.select('#scoretable > tr[style*="height:18px;"]')
print(len(scores))

>>>613

answered Feb 27 '17 at 09:41

Zroq

8,002
3
26
37

Aditya Mishra · Answer 2 · 2017-02-27T10:02:50.687

2

Try this -

req_url = 'http://www.xscores.com/soccer/livescores/25-02'
request = requests.get(req_url)
html=request.text
soup = BeautifulSoup(html, "html5lib")
scoretable=soup.find('tbody',id='scoretable')
scores=scoretable.find_all('tr')
len(scores)
>617

edited Feb 27 '17 at 10:02

answered Feb 27 '17 at 09:41

Aditya Mishra

194
1
13

umutto · Answer 3 · 2017-03-13T01:18:09.567

This line only finds rows with 'height:18px; style.

scores = soup.find_all('tr', {'style': 'height:18px;'}, limit=None)

If you look at the page source and search for "height:18px;" you'll see 50 matches. But if you search for height:18px; without the quotes you'll see 613 matches.

You need to edit that line to find rows that has height:18px; style (and other values). You can pass a regex as style argument according to documentations, maybe something like this:

soup.find_all('tr', style = re.compile('height:18px'), limit=None)

BeautifulSoup find_all limited to 50 results?

3 Answers3