Too long xpath with DOMXpath query/evaluate return nothing

Question

I am using PHP to retrieve content for a given URL and XPATH. I use DOMDocument / DOMXPath (with query or evaluate).

For small xpath, I obtain correct result, but for longer xpath, it does not work. (And this xpath seems to be good (I obtained them with Xpather (firefox plugin) and re-test them with YQL).

Do you have any advice on this curious trouble ?

Example of code:

$doc = new DOMDocument();
$myXMLString = file_get_contents('http://stackoverflow.com/questions/4097230/too-long-xpath-with-domxpath-query-evaluate-return-nothing');
@$doc->loadHTML($myXMLString); //@ to suppress warnings 
                               //(good for not ending markup)
$xpath = new DOMXPath($doc);

$fullPath ="/html/body/small/path"; //it works
//$fullPath = "/html/body/full/path/with/lot/of/markup";//does not works
$entries = $xpath->query($fullPath);
//or ->evalutate($fullPath) (same behaviour)
//$entries return DOMNodeList (empty for a long path query, 
//                             correct for a small path query)

I test with attribute restriction, but is seems to not change (with small xpath it works, with longer it do not works more)

Example : for this current page:

$fullPath = "/html
              /body
               /div[4]
                /div[@id='content']
                 /div[@id='question-header']
                  /h1
                   /a";//works (retrieve the question title)
$fullPath = "/html
              /body
               /div[4]
                /div[@id='content']
                 /div[@id='mainbar']
                  /div[@id='question']
                   /table
                    /tbody
                     /tr[2]
                      /td[2]
                       /div[@id='comments-4097230']
                        /table
                         /tbody
                          /tr[@id='comment-4408626']
                           /td[2]
                            /div
                             /a"; //does'nt work 
                                  //(should retrieve 'gaby' from comment)

---

Edit:

I test with SimpleXML lib, and I have exactly the same behavior (good result for small query, nothing for long query).

---

Edit 2:

I also cut the longest xpath by deleting some first element and it works. BTW I really do not understand why a full correct xpath does not work.

Answer 1

Let's go through this step by step:

Step 1: replicating the error.

After verifying that the XPath will indeed not return a result, I wrote a little script to see how deep the XPath will go before it breaks

foreach (explode('/', $fullPath) as $segment) {
    $xpath .= trim($segment);
    echo '-------------------------------------------', PHP_EOL,
         'Trying: ', $xpath, PHP_EOL,
         '-------------------------------------------', PHP_EOL;
    echo $xp->evaluate("string($xpath)"), PHP_EOL;
    $xpath .= '/';
}

The last thing it will return a result for is

/html/body/div[4]/div[@id='content']/div[@id='mainbar']/div[@id='question']/table

---

Step 2: checking the markup

So I checked the markup returned by DOMDocument::saveHTML() to see what it looks like and there was no <tbody> (reformatted for readability):

<div id="question">
    <div class="everyonelovesstackoverflow" id="adzerk1"></div>
        <table>
            <tr><td class="votecell">

I then checked this very page to see if it was DOM throwing it away or if it really does not exist. It wasn't there. Apparently, Firebug inserts it, which would explain why you got the result with XPather (but not why you got it with YQL):

Step 3: proofchecking and conclusion

I removed the <tbody> from the XPath and reran the script. No problems. Returns "Gaby".

While I suspected a bug in Firebug first, Alejandro commented this would happen in IE's DeveloperTools, too. I then suspected this to be added by JavaScript but could not verify that. After some more research Alejandro pointed me to Why does firebug add <tbody> to <table>? - it's actually neither Firebug nor JavaScript though, but the browser's themselves.

So to modify my conclusion:

Dont trust markup you see rendered in the browser, because it may be modified by the browser or other technologies. DOM will only download what is is served directly. If you run into similar issues again, you now know how to approach it though.

---

Some additional sidenotes

Unless you need to modify the markup before feeding it to DOM, you do not have to use file_get_contents() to load the content. You can use DOM's loadHTMLFile():

$dom->loadHTMLFile('http://www.example.com/foo.htm');

Also, the proper way to suppress errors is to tell libxml to use it's internal error handler. But instead of handling the errors, you simply clear them. This will only affect errors relating to libxml, e.g. parsing errors (as opposed to all PHP errors):

libxml_use_internal_errors(TRUE);
libxml_clear_errors();

Finally, xPath queries can be done in relation to a context node. So while the long XPath is efficient in terms of lookup time, you could simply use getElementById() to get the deepest known node and then use an XPath against it.

In other words:

libxml_use_internal_errors(TRUE);
$dom = new DOMDocument;
$dom->loadHTMLFile('http://www.example.com/foo.htm');
libxml_clear_errors();
echo $xp->evaluate(
    'string(td[2]/div/a)', 
    $dom->getElementById('comment-4408626'));

will return "Gaby" as well.