Extract HTML from a site using PHP

Question

This is the site which I am referring.

I have search through stackoverflow and tried various suggested php methods like file_get_contents() and readfile() method but it cannot retrieve the table value from the site.

i tried to view the source from the page and I could not locate the table value as well. I tried looking for iframe src but to no avail.

Not sure if there is any method which I can use to retrieve such value from the site?

Please advise.

complex857 · Accepted Answer · 2014-03-09T08:05:16.510

2

The table's html seems to be generated on the client side (in your browser) with javascript, so it won't show up in the server's response in the way you see it in the browser (you can try disabling javascript and check the site). You can either:

Switch technology, and use some kind of remote controller browser like phantomJS
You can use try to use their raw data. Just open up your browser's developer tools (usually F12) and check what URL's are fetched. You might need to try to analyze the site's javascript code to make sense of these. You should see something like this:

In both cases, check with the site's owners if they are OK with this kind of use (read their data use policy if they have one or just e-mail them), most site owners are not exactly too happy this kind of crawling.

edited Mar 09 '14 at 08:05

answered Mar 09 '14 at 07:59

complex857

20,425
6
51
54

I have tried to disable javascript and refresh the site and indeed the table is not shown. This is quite new to me as usually the most I encountered is site using iframe rather than javascript. For point number 2, how did you manage to retrieve the GET code with the xml address? Is it achievable via Firefox firebug? I tried using firebug and do a search but can't seems to find the source. (Not really an expert in firebug). Please advise. – Withhelds Mar 09 '14 at 09:00
I've used firebug, the image is just a part of the console panel's output (you might need to enable the "show XMLHttpRequests option) or just look at the net panel and hit "XHR" (the "all" can be overwhelming) on the top row. Just refresh the page with the firebug panel already open. – complex857 Mar 09 '14 at 09:22
I saw it, thanks for your explanation. I will also explore on phantomJS. Will mark your answer as accepted answer. – Withhelds Mar 09 '14 at 12:42

score 1 · Answer 2 · answered Mar 09 '14 at 07:58

Use the logic of curl, please refer this example

<?php 
        // create curl resource 
        $ch = curl_init(); 

        // set url 
        curl_setopt($ch, CURLOPT_URL, "example.com"); 

        //return the transfer as a string 
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 

        // $output contains the output string 
        $output = curl_exec($ch); 

        // close curl resource to free up system resources 
        curl_close($ch);      
?>

I tried this and echo out but still can't see the table output. Did I miss out anything? — Withhelds, Mar 09 '14 at 08:11

Extract HTML from a site using PHP

2 Answers2