0

I wrote PHP code to extract 100 data from a site, but it extract 20 data. Because after scrolling, other datas appear on the site. How can the code make scroll? What should I add to my code?

I used the following when writing the code:

file_get_contents()

preg_match_all()
Obsidian
  • 3,719
  • 8
  • 17
  • 30
Aycan Candar
  • 63
  • 1
  • 9
  • I don't have any experience with PHP, but if `file_get_contents` simply download page content it is nearly impossible: you simply parse HTML page while dynamic entries are loaded by JS (which require stuff like real browser). – val - disappointed in SE Jun 16 '19 at 23:02

1 Answers1

1

If the website has a javascript scrolling effect that load more content, you can't easily do it in PHP.

One possibility is to inspect the ajax call and simulate that in PHP to obtain the other results. To do so you could open your browser developer tools, go to the network tab, and filter by xhr. Now when you scroll and it loads the content, you can see in the developer tools the ajax call that is being made, with the link, the method and the parameters passed. Making the same call within your PHP script will obtain the same data.

You can see this answers to obtain more detailed instructions:

Request Monitoring in Chrome

In Firefox, how do I see HTTP request headers? (where in web console?)

salvatore
  • 511
  • 4
  • 11
  • I don't understand "One possibility is to inspect the ajax call and simulate that in PHP to obtain the other results" – Aycan Candar Jun 17 '19 at 09:53