Selenium: Scroll to the bottom of a dynamically loading div

Question

I'm trying to use selenium to scrape a webpage, and one of my elements dynamically loads a finite amount of content (IE not like twitter where you can just keep scrolling) when I scroll to the bottom of it.

Now I could simply do something like

var scrollbar = document.getElementsByClassName("dataTables_scrollBody")[0]
var last = -1;
var curr = scrollbar.scrollHeight;
while (curr != last) {
    last = curr;
    scrollbar.scrollTop = curr;
    curr = scrollbar.scrollHeight;
}

And in fact, that's what I'm doing. However, the actual load operation on the data table takes so long that this script exits by the time the load finishes, meaning I only ever get halfway down the scrollbar.

What's a good way to make sure that I've scrolled all the way to the bottom of the scroll bar?

This question is different from Scroll to bottom of div? because I don't have access to the raw HTML, and it's different from Scroll Automatically to the Bottom of the Page because my page is dynamically loading content.

Perhaps I could force an ajax load of all the content in the panel, but I don't know how to do that, and I don't want to wade through miles of minified code to figure out how the page owner does it.

Is there anyway to insert a "pause" in your while loop so you can slow it down? — GetSet, Mar 07 '22 at 20:10
No, javascript only seems to have asynchronous waits/pauses. — Jakob Lovern, Mar 07 '22 at 20:12
What about a nested loop that does nothing but increment a number as a makeshift pause? — GetSet, Mar 07 '22 at 20:12
Perhaps. I might be able to use a ResizeObserver on the scrollable element instead and just get it to scroll every time it resizes. — Jakob Lovern, Mar 07 '22 at 20:14
You can make a function for sleep. Hope this link will help you https://stackoverflow.com/questions/951021/what-is-the-javascript-version-of-sleep — KunduK, Mar 07 '22 at 20:15

Jakob Lovern · Accepted Answer · 2022-11-28T20:33:40.940

Here's an ugly solution:

scroll_elem = document.getElementsByClassName("dataTables_scrollBody")[0];
table_elem = document.getElementsByClassName("dataTable")[0];
function scrollToBottom() {
    scroll_elem.scrollTop = scroll_elem.scrollHeight;
};
new ResizeObserver(scrollToBottom).observe(table_elem)

Basically how this works is we have an object watching the element we want to scroll to the bottom of. If and when it resizes, it instantly scrolls to the bottom of the new window. This works well enough, then you could use the sleep function KunduK recommended in comments to wait for the process to complete

Selenium: Scroll to the bottom of a dynamically loading div

1 Answers1