Webscraping Beautiful Soup - Table ends prematurely?

Question

I'm trying to learn webscraping in python using BeautifulSoup. I've come a bit stuck. I'm trying to scrape this wikipedia page;

https://en.wikipedia.org/wiki/List_of_MPs_elected_in_the_United_Kingdom_general_election,_2015

Specifically the table of the list of MPs elected at the general election.

So far my code is;

main_page = 'https://en.wikipedia.org/wiki/List_of_MPs_elected_in_the_United_Kingdom_general_election,_2015'

page= requests.get(main_page)
soup = BeautifulSoup(page.text, "lxml")
table = soup.find("table", class_="wikitable sortable")

When I try to view the variable table, it seems to stop prematurely, with the following being the last row in the table;

<tr>\n<td><a href="/wiki/Carmarthen_East_and_Dinefwr_(UK_Parliament_constituency)" title="Carmarthen East and Dinefwr (UK Parliament constituency)">Carmarthen East and Dinefwr</a></td>\n<td style="background-color: #008142"></td>\n<td><a href="/wiki/Plaid_Cymru" title="Plaid Cymru">Plaid Cymru</a></td>\n<td style="background-col"></td></tr>

However, the table on wikipedia goes on for hundreds more rows. Does anyone know why i'm not able to pick up the rest of the data?

I get 650 rows with `` cells in them, the last is `York Outer`. The table is not truncated. — Martijn Pieters, Jul 10 '16 at 12:43
In other words: it works just fine for me, we can't reproduce your issue with just the code you posted. — Martijn Pieters, Jul 10 '16 at 12:45
Hi, thanks for trying it! I just managed to solve the problem by changing the parser from lxml to html.parser ... now it seems to work — Noel, Jul 10 '16 at 13:14

Webscraping Beautiful Soup - Table ends prematurely?

0 Answers0