I've been looking for a solution to scrape data from a table where the html dynamically changes as you scroll through the table. The html will only show the data that is actually displayed on screen in the browser.
What I have done so far is use Selenium to deal with the rendering of the html code. I have thought about trying to scroll through the table and collect the data in increments but 1) I cannot seem to actually scroll through the table using selenium as class names have spaces 2) I feel there must be a better way to grab all the data in one go.
Edit: This is not a duplicate question as suggested in the comments. I have no problem obtained html code rendered by JavaScript. But the JavaScript for the table I am referring to will only be executed if it needs to be i.e. if the table is scrolled and hence needs to be displayed on screen. What this means if I scroll through the table, the html code for the newly displayed rows will appear but the code for the previously displayed rows will disappear.
Any help would be much appreciated.
