0

I'm trying to scrape a webpage where the content I want loads after the DOM completes. The new content is fetched through AJAX calls.

So the fetched content isn't available in the page source. I can see when inspecting the page.

When I use cURL it doesn't find the elements on the page. What is the best method to get this content?

I'm trying to use PhantomJS for this, but I'm not sure if that can do it either.

Thanks.

fractal5
  • 2,034
  • 4
  • 29
  • 50
  • Possible duplicate of [How do you scrape AJAX pages?](http://stackoverflow.com/questions/260540/how-do-you-scrape-ajax-pages) – Rohin Gopalakrishnan Nov 13 '16 at 06:44
  • You can examine the request made by AJAX (e. g. in Chrome developer tools on Network tab), reproduce the same XHR, and retrieve the data not via DOM, but parsing the response. – omegastripes Nov 13 '16 at 10:02
  • I found the AJAX call but the parameters supplied are in POST form. Is there a way to find the info being sent through POST? – fractal5 Nov 13 '16 at 17:01

0 Answers0