unable to scrape content from a website

Question

I am trying to scrap some content from a website but the code below is not working(not showing any output). here is the code

$url="some url";
$otherHeaders="";   //here i am using some other headers like content-type,userAgent,etc
some curl to get the webpage
...
..
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$content=curl_exec($ch);curl_close($ch);

$page=new DOMDocument();
$xpath=new DOMXPath($page); 
$content=getXHTML($content);  //this is a tidy function to convert bad html to xhtml 
$page->loadHTML($content);    // its okay till here when i echo $page->saveHTML the page is displayed

$path1="//body/table[4]/tbody/tr[3]/td[4]";
$path2="//body/table[4]/tbody/tr[1]/td[4]";

$item1=$xpath->query($path1);
$item2=$xpath->query($path2);

echo $item1->length;      //this shows zero 
echo $item2->length;      //this shows zero

foreach($item1 as $t)
echo $t->nodeValue;    //doesnt show anything
foreach($item2 as $p)
echo $p->nodeValue;    //doesnt show anything

i am sure there is something wrong with the above xpath code. the xpaths are correct. I have checked the above xpaths with FirePath (a firefox addon). I know i am missing something very silly here but i cant make out. Please help. I have checked similar code for scraping links from Wikipedia(definitely the xpaths are different) and it works nicely. So i dont understand why the above code does not work for the other URLs. I am cleaning the HTML content with Tidy so i dont there is a problem with xpath not geeting the HTML right? i have checked the length of the nodelist after $item1=$xpath->query($path1) which is 0 which means something is going wrong with $xpath->query because the xpaths are correct as i have checked with FirePath I have modified my code a bit as pointed out and used loadXML instead of loadHTML. but this gives me error as Entity 'nbsp' not defined in Entity so i used the libxml option LIBXML_NOENT to substitute entities but still the errors remain.

Answer 1

Yes, you are missing something very basic: It's XHTML, so you must register (and use!) the proper namespace before you can expect to get results.

$xpath->registerNamespace('x', 'http://www.w3.org/1999/xhtml');

$path1="//x:body/x:table[4]/x:tbody/x:tr[3]/x:td[4]";
$path2="//x:body/x:table[4]/x:tbody/x:tr[1]/x:td[4]";

$item1=$xpath->query($path1);
$item2=$xpath->query($path2);

Answer 2

It seems that the problem is somehow related to XPath and namespaces. Php manual revealed an interesting user comment

If you've registered your namespaces, loaded your XHTML, etc., into your XPath's DOMDocument object and still can't get it to work, check to make sure you haven't used the DOMDocument's loadHTML() or loadHTMLFile() function. For XHTML always use the XML versions, otherwise your XPath will never, ever work.

Your code uses loadHTML()

$content=getXHTML($content);  //this is a tidy function to convert bad html to xhtml 
$page->loadHTML($content);    // its okay till here when i echo $page->saveHTML the page is displayed

HTML is not namespace aware so loadHTML() might not set the namespaces on the elements of the document object even though the original document (or the XHTML outputted by Tidy) had them.

Because you use Tidy to convert the document to XHTML, I guess you could safely use loadXML() without running into parsing errors. Note that it will require that the input is well-formed XML. Also it might not be aware of HTML predefined entities like   and if that is the case, it can't replace the entities with their correct character values. If such problem arises, try setting different options for loadXML().

Answer 3

I have heard that FireFox adds a tbody element if such isn't present.

In addition to or independently of @Tomalak's advice, try the XPath expressions with the /tbody location step removed.

Also, use another tool as the XPath Visualizer to construct correct XPath expressions and see immediately what they are selecting.

Answer 4

This question reminds me that a lot of times the solution to a problem lies in simplicity and not complications. i was trying namespaces,error corrections,etc but the solution just demanded close inspection of the code. the problem with my code was the order of loadHTML() and xpath initialization. initially the order was

$xpath=new DOMXPath($page);
$page->loadHTML($content);

by doing this i was actually initializing xapth on an empty document. now reversing the order by first loading the dom with the html and then initializing the xpath i was able to get the desired results. Also as suggested that by removing the tbodyelement from xpath as firefox automatically inserts it. so the correct xpath should be

$path1="//body/table[4]/tr[3]/td[4]";
$path2="//body/table[4]/tr[1]/td[4]";

thanks to everyone for their suggestions and bearing this.

Answer 5

(Try the following both in combination with and separately from the other answers, as they are other possible caveats.)

If your XPath isn't working, try applying just parts of it to make sure you are indeed following the right path. So do something like:

$path1="//body";
$item1 = $xpath->query($path1);

foreach ($item1 as $t) {
    // to see the full XML of the returned node, as the nodeValue may be empty
    echo $t->ownerDocument->saveXML($t); 
}

Then keep increasing your XPath to the location you want.

Also, if you find that nodeValue and textContent of your nodes are empty, you should make sure that you are loading into the DOMDocument with the correct encoding (e.g. if the cURL response returns UTF-8, you'll need to pass 'UTF-8' as the second parameter when constructing your DOMDOcument).