Scroll/Load dynamic table in with selenium webdriver

Question

I am trying to get all rows from dynamic table which is not fully loaded. It has nearly 2k rows so that I need to scroll down to load table then I need to get all elements from xpath. Here is my code block:

    try:
    elem = driver.find_element_by_xpath("//table[@class = 'ng-scope ng-table']")
    for _ in range(50):
        driver.execute_script("arguments[0].scrollIntoView();", elem)
        driver.implicitly_wait(0.2)
    except Exception as e:
        print ("Error scrolling down web element = " + str(e))
        driver.quit()

    # To get all elements in table (It only takes first 25 since others could not be loded without scrolling)
    elements = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class = 'scrolly-table']//tr[@class = 'base ng-scope']")))

But its action is putting the table top of screen. Even i try to manually scroll it while putting some sleep points, it again sctoll to top of table. Neither gives error nor scrolls the table.

All suggestions are welcome

Normally lazy loading web pages use apis to populate so most likely you can use a simple request to grab all the data. Look into the Network tab in the developer tools and reload the page and see if you can see if there's anything like that. Otherwise you can use the following to load pages dynamically. — Arundeep Chohan, Jan 21 '21 at 03:33
https://stackoverflow.com/questions/62600288/how-to-handle-lazy-loaded-images-in-selenium is most likely what it is doing. — Arundeep Chohan, Jan 21 '21 at 03:36
Actually it is not the same, because scrolling down the page to the end does not mean to scrolling down the table. So that my table which includes my desired values still not loaded. Instead of scrolling down the page, I need to scroll down to table to the end. But it has nearly 2k rows. — hyorman, Jan 21 '21 at 08:01

score 0 · Answer 1 · edited Aug 24 '21 at 14:57

This is what worked for me while I was trying to get elements on a scroll/load table :

1- click the table to select it:

driver.find_element_by_xpath('//*[@id="table_id"]/tbody').click()

2- scroll down the whole page:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

3- for my case, tr elements were grouped in tbody by 30. So I based my code on a loop that gets tbody's trs, then scroll in the table:

trs = []
x = 1
while x <= tbody:
        lines = driver.find_elements_by_xpath('//*[@id="table_id"]/tbody[%s]/tr' % x)
        for line in lines:
            tr = CDMTransaction()
            tr.label = line.find_element_by_xpath('.//td[5]/span').text
            trs.append(tr)

        x += 1
        actions = ActionChains(self.driver)
        actions.send_keys(Keys.SPACE).perform()
        actions.send_keys(Keys.SPACE).perform()
        time.sleep(1)

Scroll/Load dynamic table in with selenium webdriver

1 Answers1