-1

I'm trying to identify the exact mechanism youtube uses to lazy load a page. For example, I want to scrape all links from a user's page with wget or something similar, but the page is incrementally (lazy) loaded, and does not return all content.

I'm not a web pro, but am a good coder. I have to admit JS is hard to follow in practice. Is this form of lazy loading based in the browser or in the script?

Watching the network activity in the developer tools window show the images being loaded as the page is scrolled down, but I can't tell what mechanism is causing this.

bbhack
  • 11
  • 2

1 Answers1

0

These pages load their content with JavaScript as you scroll down, so to scrape it you need to track the AJAX calls that are made or use a framework to do it, like PhantomJS. You can also try to build your robot, as you said you're a good programmer, so you can try to follow some ideas here and here.

Felipe G.
  • 154
  • 1
  • 9