1

I am trying to scrape a (very long) webpage for some content.
I am using Python's BeatifulSoup library, but am not limited to Python. The issue I am facing is that the page only loads a small subset of the content, and, as soon as the user scrolls to the top/bottom of the page, more content is loaded.

There is a javascript function that is called when the scroll-bar hits the top/bottom, triggering the server to load more data. Is there a way for me to programmatically scroll through the page, or call the function that gets the page to load more data, so I have all the content available? I know I could just use VBScript to control the window and scroll, but ideally I am looking for a more elegant solution that allows to me to extract the data and scroll the page at the same time.

samuelschaefer
  • 614
  • 2
  • 10
  • 26

1 Answers1

0

You can check following answer OR you can check the net panel and call the same url or method which gets called when page hits the bottom of the page

Vinayak Sakhare
  • 768
  • 1
  • 7
  • 14
  • This answer specifically tells about the JavaScript part but OP needs pythonic answer of getting html out from the page. – Rajan Chauhan Jul 31 '17 at 22:08