0

I'm trying to extract all the contents of a table from a website using Python and Selenium. However, not all the table contents are visible at once. The table updates its HTML script and loads new contents only when I move the mouse to the table and scroll down using the mouse scroller. Unfortunately, the website does not have a scroll-down button to automate this process.

How can I use Selenium in Python to simulate the scrolling action and extract all the dynamically loaded table contents? Any suggestions or code examples would be highly appreciated!

Iam a beginner to selenium and I have used the codes that are available online but none worked. Thanks in advance for the help.

Bhawesh Kumar Singh
  • 1
  • Does this answer your question? [How can I scroll a web page using selenium webdriver in python?](https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python) – Rolandas Ulevicius Jul 25 '23 at 07:00

1 Answers1

0

First, you need to find a unique identifier of the table or a way to locate the table. This can be done by opening developer tools via any browser.

Next, you would want to simulate user behavior (in your case scroll and move the mouse on the table).

Finally, you will extract the data you need.

For example, if the table's id is 'data' then you would want to tell Selenium driver to scroll the table into view using javascript:

driver.execute_script("document.getElementById('data').scrollIntoView();")

If the unique identifier is a class then use getElementByClass and etc. You can find more functions online.

Next, JavaScript doesn't let us control the mouse, so we will use Selenium to do that.

Import ActionChains to perform actions:

from selenium.webdriver import ActionChains

Then move the mouse:

element = driver.find_element(By.ID, 'data')
action = ActionChains(driver)
action.move_to_element(element)
action.perform()

And finally, extract the data you need! This also can be done in different ways, this example gets the inner HTML:

element.get_attribute('innerHTML')

You can find more data here: https://selenium-python.readthedocs.io/api.html

DenisG
  • 46
  • 4