selenium phantomjs thead there but tbody missing

Question

While scraping this page:

https://www.hkex.com.hk/Products/Listed-Derivatives/Equity-Index/Hang-Seng-Index-(HSI)/Hang-Seng-Index-Futures?sc_lang=en#&product=HSI

in google chrome key F12, I see the xpath

  t//*[@id="equity_future"]

has a thead and a tbody. The tbody is available.

However, inside python3 debugger, with

wdriver = webdriver.PhantomJS()
wdriver.get(url)
soup = BeautifulSoup(wdriver.page_source,"lxml")

I do see the thead children but the tbody appears empty

<tbody>
</tbody>

Any ideas?

undetected Selenium · Answer 1 · 2019-05-05T15:38:03.420

0

Using only Selenium if you extract the page_source you can find all the <tbody> tags as follows:

Code Block:

driver = webdriver.PhantomJS(executable_path=r'C:\WebDrivers\phantomjs.exe')
driver.get("https://www.hkex.com.hk/Products/Listed-Derivatives/Equity-Index/Hang-Seng-Index-(HSI)/Hang-Seng-Index-Futures?sc_lang=en#&product=HSI")
print(driver.page_source)

Console Output Snippet 1:

<tbody>
<tr>
    <td class="ls">Last Traded</td>
    <td class="vo">Volume</td>
    <td class="oi">Prev.Day Open Interest</td>
</tr>
</tbody>

Console Output Snippet 2:

<tbody>
<tr>
    <td class="se">Prev.Day Settlement Price</td>
    <td class="vo">Volume</td>
    <td class="oi">Prev.Day Open Interest</td>
</tr>
</tbody>

edited May 05 '19 at 15:38

answered May 05 '19 at 15:06

undetected Selenium

183,867
41
278
352

Printing just driver.page_source still doesn't show tbody rows like May-19. Also empty \n \n – MMM May 05 '19 at 17:47
@MMM Which `tbody` rows do you want to see exactly? – undetected Selenium May 05 '19 at 18:16
Here is an example row I see with Chrome May-1929,863+20529,86329,862
29,87929,55429,875
29,412147,228121,569 – MMM May 05 '19 at 20:08
DebanjanB They are rows which have numbers in them. I can see them with the browser but not with PhantomJS. Do you agree? – MMM May 08 '19 at 06:48

selenium phantomjs thead there but tbody missing

1 Answers1