0

I am currently working on a small web scraping script with Python and Selenium.

I am trying to get some information from a table, which has in inspection mode a certain ID. However, when I open the page as raw HTML (which I did after not being able to locate that table using neither xpath or css_selector), the table does not have the mentioned ID.

How is that possible?

For better explanation:

This is what it looks like in inspection mode in my browser

<table id='ext-gen1076' class='bats-table bats-table--center'>
[...]
</table>

And this is what it looks like when I open the page as raw HTML file

<table class='bats-table bats-table--center'>
[...]
</table>

How is it possible, that the ID is simply disappearing? (JFI, this is my first question so apologies for the bad formatting!).

Thanks in advance!

devcayed
  • 37
  • 6
  • Check this post its similar https://stackoverflow.com/questions/13070190/is-the-html-shown-via-view-source-different-from-the-html-shown-in-firebug-d – mk_ Feb 29 '20 at 11:43

2 Answers2

2

The reason is, the ID was added during the runtime.

Archer
  • 1,062
  • 1
  • 13
  • 32
0

The value of the id attribute i.e. ext-gen1076 contains a number and is clearly dynamically generated. The prefix of the value of the id attribute i.e. ext-gen indicates that the id was generated runtime using Ext JS.

Ext JS

Ext JS is a JavaScript framework for building data-intensive, cross-platform web and mobile applications for any modern device.

This usecase

Possibly you have identified the <table> element even before the JavaScript have rendered the complete DOM Tree. Hence the id attribute was missing.

Identifying Ext JS elements

As the value of the id attribute changes i.e. dynamic in nature you won't be able to use the complete value of the id attribute and can use only partial value which is static. As per the HTML you have provided:

<table id='ext-gen1076' class='bats-table bats-table--center'>
[...]
</table>

To identify the <table> node you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table[id^='ext-gen']")))

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[starts-with(@id,'ext-gen')]")))

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

However, there would be a lot other elements with id attribute starting with ext-gen. So to uniquely identify the <table> element you need to club up the class attribute as follows:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.bats-table.bats-table--center[id^='ext-gen']")))

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='bats-table bats-table--center' and starts-with(@id,'ext-gen')]")))

Reference

You can find a relevant detailed discussion in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352