How to scrape data which have auto-scroll and lazy loading

Asked Aug 19 '19 at 10:29

Active Aug 19 '19 at 11:36

Viewed 352 times

I want to scrape some data from a website. (I am using Rails and Nokogiri.) The site which I want to scrape is having lazy loading with a scroll event. But I have a problem with the url of ajax request.

The site uses a complex request : load_more?id=1837373&lastPostid=987984545 instead of a regular request like ?page=1.

Is it possible to scroll down 2 times and fetch all data?

Thoughts?

asked Aug 19 '19 at 10:29

Oğuz Çiçek

Some examples of the HTML DOM ? Example: find the total count of visible elements per page, iterate them and on the latest, send scroll event ? – Infern0 Aug 19 '19 at 10:31
the xhr request is not like ?page=1 to easily iterate.. – Oğuz Çiçek Aug 19 '19 at 11:01

How to scrape data which have auto-scroll and lazy loading

0 Answers0