I am trying to scrape a table off the html pic linked below.
The table I am looking for is in the tbody that I've highlighted in blue, however, when I use Beautifulsoup to scrape this html document, I can only get information up until ('div', 'class': 'sic_localNav'). Somehow, Beautifulsoup does not pick up all the information,(and the table) within the div class. - I have drawn an arrow to denote where this div class is.
a = bs4.BeautifulSoup(response.text, 'lxml')
b = a.find('div', attrs={'class': 'sic_localNav'})
print(b)
<div class="sic_localNav">
<ul class="sic_section {default_type: 'insider_latest', default_market: 'sgx'}" id="nav_l3">
<li class="nav_l3_tabs {type: 'insider_latest'}">
<h3>
Latest Insider Trades
</h3>
</li>
<li class="nav_l3_tabs {type: 'insider_significant_sales'}">
<h3>
Significant Sales
</h3>
</li>
<li class="nav_l3_tabs {type: 'insider_significant_purchases'}">
<h3>
Significant Purchases
</h3>
</li>
<li class="nav_l3_tabs {type: 'insider_search'}">
<h3>
Search
</h3>
</li>
</ul>
<img alt="loading..." class="sic_icon_loadingLarge" height="32" id="sic_insider_loading_indicator" src="/images/icon_loadingLarge.gif" style="display: none;" width="32"/>
<div class="sic_innerContainer" id="sic_insider">
</div>
</div>
Any thoughts on why Beautifulsoup is not picking up the tbody? I've also checked, this tbody is not within an iframe or anything like that.
Any thoughts?
Sorry that the image is slightly blurry. It is not that your eyes are tired..
