1

How to scrape infinite scroll page in Reddit and add limit scroll?

URL: https://www.reddit.com/r/teslamotors/new/

What I am aiming is to get the items in the firstpage but I am just getting some few items.

What I have tried is to go the developers tool and check the XHR but I am getting different approach and can't find page there.

This is what I am using to fetch the item.

response.xpath("//a[@class = 'b5szba-0 fMXvWW']/@href").getall()

Please help thank you so much

Christian Read
  • 135
  • 11
  • I do hope no one helps you with this. https://www.redditinc.com/policies/user-agreement-may-25-2018-2 - Access, query, or search the Services with any automated system, other than through our published interfaces and pursuant to their applicable terms. However, we conditionally grant permission to crawl the Services for the sole purpose of and solely to the extent necessary for creating publicly available searchable indices of the materials subject to the parameters set forth in our robots.txt file. – Trent May 20 '19 at 02:41
  • 1
    "However, we conditionally grant permission to crawl the Services for the sole purpose of and solely to the extent necessary for creating publicly available searchable indices of the materials subject to the parameters set forth in our robots.txt file". I think what I am asking is ok sir @Trent – Christian Read May 20 '19 at 02:56
  • 1
    Check this out: https://www.youtube.com/watch?v=EelmnSzykyI – Thiago Curvelo May 21 '19 at 01:58
  • Please, see https://stackoverflow.com/q/8550114/939364 – Gallaecio May 21 '19 at 14:32

0 Answers0