Stream parse 4 GB XML file in PHP

Question

I'm trying and need some help doing the following:

I want to stream parse a large XML file ( 4 GB ) with PHP. I can't use simple XML or DOM because they load the entire file into memory, so I need something that can stream the file.

How can I do this in PHP?

What I am trying to do is to navigate through a series of <doc> elements. And write some of their children to a new xml file.

The XML file I am trying to parse looks like this:

<feed>
    <doc>
        <title>Title of first doc is here</title>
        <url>URL is here</url>
        <abstract>Abstract is here...</abstract>
        <links>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
       </link>
    </doc>
    <doc>
        <title>Title of second doc is here</title>
        <url>URL is here</url>
        <abstract>Abstract is here...</abstract>
        <links>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
            <sublink>Link is here</sublink>
       </link>
    </doc>
</feed>

I'm trying to get / copy all the children of each <doc> element into a new XML file except the <links> element and its children.

So I want the new XML file to look like:

<doc>
    <title>Title of first doc is here</title>
    <url>URL is here</url>
    <abstract>Abstract is here...</abstract>
</doc>
<doc>
    <title>Title of second doc is here</title>
    <url>URL is here</url>
    <abstract>Abstract is here...</abstract>
</doc>

I would greatly appreciate any and all help in streaming / stream parsing / stream reading the original XML file and then writing some of its contents to a new XML file in PHP.

Check out the XMLReader class: http://www.php.net/manual/en/intro.xmlreader.php It's a streaming parser. I'm reading your question more deeply right now to see if I can help with more specific answers. — DeeDee, Aug 29 '13 at 18:39
@DeeDee I had heard of XMLReader, but didn't know how to use it. Thank you for the help! — Django Johnson, Aug 29 '13 at 19:30
Sure! It's not too heavily used, as evidenced by the dearth of comments in the official documentation. I myself haven't used it in a very long time. Can you let me know how my code works? If it doesn't work immediately we can collaborate and figure out what's up. — DeeDee, Aug 29 '13 at 19:32
@DeeDee Yes, sure thing. I will try it out later tonight and let you know if it works or any errors I get. Thank you for helping me and for being willing to collaborate until it is solved :-) — Django Johnson, Aug 29 '13 at 19:34
And.. why was this closed? I'm pretty sure the question is on topic for php, xml, xml-parsing, large files.. wtf — That Realty Programmer Guy, Apr 15 '14 at 15:15

DeeDee · Answer 1 · 2013-08-31T22:06:18.407

4

Here's a college try. This assumes a file is being used, and that you want to write to a file:

<?php

$interestingNodes = array('title','url','abstract');
$xmlObject = new XMLReader();
$xmlObject->open('bigolfile.xml');

$xmlOutput = new XMLWriter();
$xmlOutput->openURI('destfile.xml');
$xmlOutput->setIndent(true);
$xmlOutput->setIndentString("   ");
$xmlOutput->startDocument('1.0', 'UTF-8');

while($xmlObject->read()){
    if($xmlObject->name == 'doc'){
        $xmlOutput->startElement('doc');
        $xmlObject->readInnerXML();
        if(array_search($xmlObject->name, $interestingNodes)){
             $xmlOutput->startElement($xmlObject->name);
             $xmlOutput->text($xmlObject->value);
             $xmlOutput->endElement(); //close the current node
        }
        $xmlOutput->endElement(); //close the doc node
    }
}

$xmlObject->close();
$xmlOutput->endDocument();
$xmlOutput->flush();

?>

edited Aug 31 '13 at 22:06

answered Aug 29 '13 at 19:08

DeeDee

2,641
2
17
21

What was in your latest edit? I can't tell the difference between this , current version and the version I was reading before. – Django Johnson Aug 29 '13 at 19:32
This looks like exactly what I was looking for, thank you. I will try it out later tonight and let you know what happens. – Django Johnson Aug 29 '13 at 19:33
I closed the ` – DeeDee Aug 29 '13 at 19:33
Ah, okay. So no changes to the code. I will try it later tonight and let you know how it goes. – Django Johnson Aug 29 '13 at 19:35
I got an error when I tried to run a file with only your code in it. Here is the error I got in my error log: `PHP Fatal error: Call to undefined method XMLReader::startElement() in xml.php on line 15` – Django Johnson Aug 29 '13 at 23:13
Yep, I got my `$xmlObject` and `$xmlOutput` variables confused. Try my edit! – DeeDee Aug 30 '13 at 02:54
Ah, okay. The resulting file is created with a doctype and a unclosed ` – Django Johnson Aug 31 '13 at 14:59
Hahaha, I got in trouble with my variable names. They're very easy to confuse when working outside of an IDE. Try the latest update :) – DeeDee Aug 31 '13 at 22:06
Ah, No worries. Thank you for the help. The latest update is running without error, but all I am getting is `` elements one on each line. – Django Johnson Sep 03 '13 at 14:51

score 1 · Answer 2 · edited May 23 '17 at 12:25

For this scenario you can't afford to use a DOM parser, as you stated, it will not fit in memory due to the file size, and even if you could, it'll be slow as it first load the entire file and after that you have to iterate through it, so, for this case you should try a SAX parser (event/stream oriented), add a handler for those tag you're insterested in (doc, title, url, abstract) and for every event append the node found in the new XML file.

Here you have more information:

What is the fastest XML parser in PHP?

Here is a (not tested) sample of what the code would be:

<?php
    $file = "bigfile.xml";
    $fh = fopen("out.xml", 'a') or die("can't open file");
    $currentNodeTag = "";    
    $tags = array("doc", "title", "url", "abstract");

    function startElement($parser, $name, $attrs) {
        global $tags;

        if (isset($tags[strtolower($name)])) {
            $currentNodeTag = strtolower($name);
            fwrite($fh, sprintf("<%s>\n"));
        }
    }

    function endElement($parser, $name) {
        global $tags;

        if (isset($tags[strtolower($name)])) {
            fwrite($fh, sprintf("</%s>\n"));
            $currentNodeTag = "";
        }
    }

    function characterData($parser, $data) {
        if (!empty($currentNodeTag)) {
            fwrite($fh, $data);
        }
    }    

    $xmlParser = xml_parser_create();
    xml_set_element_handler($xmlParser, "startElement", "endElement");
    xml_set_character_data_handler ($xmlParser, "characterData");

    if (!($fp = fopen($file, "r"))) {
        die("could not open XML input");
    }

    while ($data = fread($fp, 4096)) {
        if (!xml_parse($xmlParser, $data, feof($fp))) {
            die(sprintf("XML error: %s at line %d",
                        xml_error_string(xml_get_error_code($xmlParser)),
                        xml_get_current_line_number($xmlParser)));
        }
    }

    xml_parser_free($xmlParser);
    fclose($fh);
?>

I'm getting an error with the code that I can't seem to fix. It also doesn't make sense. The error I am getting is: `PHP Parse error: syntax error, unexpected ';' in /Users/irfanm/Desktop/mamp/xml2.php on line 12'. — Django Johnson, Aug 29 '13 at 23:24

Stream parse 4 GB XML file in PHP

2 Answers2

Linked

Related