0

So i've made a few scrapers in my past but every scraper that i made is taking content from non dynamically loaded page. Basically page that is not using any ajax.

But now i am looking for solutions almost 2 days and can't find any. The page that i am trying to download and use information's from have it's content loaded from ajax. So basically when you go to that page it load content, than table that i need to get is loaded with ajax on page load.

And it looks like curl doesn't understand that page content i need is loaded dynamically so i always get boolean:false or property does not exists response.

This is the code that i am trying to get.

    $requ = require_once( get_template_directory() . '/includes/simple_html_dom.php' );
    $getfeed = curl_init();
    curl_setopt($getfeed, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
    curl_setopt($getfeed, CURLOPT_URL, 'http://wizzair.com/');
    curl_setopt($getfeed, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($getfeed, CURLOPT_CONNECTTIMEOUT, 20);
    $str = curl_exec($getfeed);
    curl_close($getfeed);

    $html = str_get_html($str);

    foreach($html->find('div.fare-finder__calendar__days__day__container') as $key => $value) {
         print_r($value);
    }

So is there a way to make it understand that content is loaded with ajax on that page so at least to make curl wait a few seconds before it loads a page ?

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
lonerunner
  • 1,282
  • 6
  • 31
  • 70
  • There are a few questions here about this type of thing. Have you seen [this one?](https://stackoverflow.com/questions/28505501/get-the-content-text-of-an-url-after-javascript-has-run-with-php) – Don't Panic Jun 28 '17 at 21:08
  • Maybe something like phantomjscloud is an option for you. – pguardiario Jun 29 '17 at 03:15
  • I see this could be good solution with something like phantomjscloud and i just tried it but it doesn't work specifically with wizzair.com website. – lonerunner Jun 29 '17 at 10:00
  • What have you tried so far? Where are you stuck? cURL usually doesn't interpret any JS code after all – Nico Haase Jun 08 '21 at 15:34

0 Answers0