Fix malformed XML in PHP before processing using DOMDocument functions

Question

I'm needing to load an XML document into PHP that comes from an external source. The XML does not declare it's encoding and contains illegal characters like &. If I try to load the XML document directly in the browser I get errors like "An invalid character was found in text content" also when loading the file in PHP I get lots of warnings like: xmlParseEntityRef: no name in Entity and Input is not proper UTF-8, indicate encoding ! Bytes: 0x9C 0x31 0x21 0x3C.

It's clear that the XML is not well formed and contains illegal characters that should be converted to XML entities.

This is because the XML feed is made up of data supplied by lots of other users and clearly it's not being validated or reformatted before I get it.

I've spoken to the supplier of the XML feed and they say they are trying to get the content providers to sort it out, but this seems silly as they should be validating the input first.

I basically need to fix the XML correcting any encoding errors and converting any illegal chars to XML entities so that the XML loads problem when using PHP's DOMDocument functions.

My code currently looks like:

  $feedURL = '3704017_14022010_050004.xml';
  $dom = new DOMDocument();
  $dom->load($feedURL);

Example XML file showing encoding issue (click to download): feed.xml

Example XML that contains chars that have not been converted to XML entities:

<?xml version="1.0"?>
<feed>
<RECORD>
<ID>117387</ID>
<ADVERTISERNAME>Test</ADVERTISERNAME>
<AID>10544740</AID>
<NAME>This & This</NAME>
<DESCRIPTION>For one day only this is > than this.</DESCRIPTION>
</RECORD>
</feed>

"How do all the other happy customers(?) deal with the data and why am I the only miserable guy" - that's a question I'd ask the provider. Can you provide an (exact) example document? — VolkerK, Feb 14 '10 at 16:22
I was wondering that myself. I've spoken to them and they told me they are having data quality issues and have told the content providers to sort it out. I'm assuming the other customers have found a way to fix the XML feed before they attempt to process it. Hence my question. — Camsoft, Feb 14 '10 at 17:13
@VolkerK I've uploaded a subset of the entire XML document as the full XML as over 42,000 lines. — Camsoft, Feb 14 '10 at 17:16

score 11 · Answer 1 · edited Jun 23 '14 at 17:36

11

To solve this issue, set the DomDocument recover property to TRUE before loading XML Document

$dom->recover = TRUE;

Try this code:

$feedURL = '3704017_14022010_050004.xml';
$dom = new DOMDocument();
$dom->recover = TRUE;
$dom->load($feedURL);

edited Jun 23 '14 at 17:36

mathielo

6,725
7
50
63

answered Feb 14 '12 at 18:10

AbhiG

336
2
4

Works with "Extra content at the end of the document in Entity" Warnings, see http://eval.in/26395 – hakre May 10 '13 at 22:00

Neel · Accepted Answer · 2010-02-17T13:33:07.487

8

Try using the Tidy library which can be used to clean up bad HTML and XML http://php.net/manual/en/book.tidy.php

A pure PHP solution to fix some XML like this:

<?xml version="1.0"?>
<feed>
<RECORD>
<ID>117387</ID>
<ADVERTISERNAME>Test < texter</ADVERTISERNAME>
<AID>10544740</AID>
<NAME>This & This</NAME>
<DESCRIPTION>For one day only this is > than this.</DESCRIPTION>
</RECORD>
</feed>

Would be something like this:

  function cleanupXML($xml) {
    $xmlOut = '';
    $inTag = false;
    $xmlLen = strlen($xml);
    for($i=0; $i < $xmlLen; ++$i) {
        $char = $xml[$i];
        // $nextChar = $xml[$i+1];
        switch ($char) {
        case '<':
          if (!$inTag) {
              // Seek forward for the next tag boundry
              for($j = $i+1; $j < $xmlLen; ++$j) {
                 $nextChar = $xml[$j];
                 switch($nextChar) {
                 case '<':  // Means a < in text
                   $char = htmlentities($char);
                   break 2;
                 case '>':  // Means we are in a tag
                   $inTag = true;
                   break 2;
                 }
              }
          } else {
             $char = htmlentities($char);
          }
          break;
        case '>':
          if (!$inTag) {  // No need to seek ahead here
             $char = htmlentities($char);
          } else {
             $inTag = false;
          }
          break;
        default:
          if (!$inTag) {
             $char = htmlentities($char);
          }
          break;
        }
        $xmlOut .= $char;
    }
    return $xmlOut;
  }

Which is a simple state machine noting whether we are in a tag or not and if not then encoding the text using htmlentities.

It's worth noting that this will be memory hungry on large files so you may want to rewrite it as a stream plugin or a pre-processor.

edited Feb 17 '10 at 13:33

answered Feb 15 '10 at 16:23

Neel

854
5
7

I've been unable to get this to work with MAMP on my Mac. It's really frustrating. – Camsoft Feb 16 '10 at 09:32
There a pure PHP library similar to Tidy called htmLawed [ http://www.bioinformatics.org/phplabware/internal_utilities/htmLawed/ ]. You may have more luck with that. – Neel Feb 16 '10 at 12:42
This looks interesting though it seems to be more about fixing XML errors. The actual errors I am having are to do with mixed encoding of the content and chars that have not been converted to XML entities. – Camsoft Feb 16 '10 at 13:58
Ideally I need some PHP to fix the encoding errors. You can see an example of the error by clicking on the example link at the bottom of the question. I've tried to to install Tidy but it won't work on my web server. – Camsoft Feb 16 '10 at 16:16
The problem is actually to do with the encoding you're loading the XML with and the solution is here: http://stackoverflow.com/questions/1269485/how-do-i-tell-domdocument-load-what-encoding-i-want-it-to-use – Neel Feb 16 '10 at 16:35
I managed to fix encoding issues by loading the XML using file_get_content and then utf8-encode but still getting `xmlParseEntityRef` errors. I'm guessing they are things like & and > < symbols in the XML. How can I convert the special chars to entities without changing the XML nodes too? – Camsoft Feb 16 '10 at 20:37
Can you post another sample as the one you've posted does not have the symbolic errors that you describe. – Neel Feb 17 '10 at 09:14
I've posted another XML example this time it has illegal chars like &, > etc. – Camsoft Feb 17 '10 at 11:38
@Neel. Seems like a lot of code to clean up the XML. I'm not to keen on having code that walks the XML tree as this particular XML file is huge and is going to be a strain on the server. My XML feed as 42,000 lines of XML and is nearly 5MB in size. – Camsoft Feb 17 '10 at 16:54
If you implement it as a stream wrapper http://uk3.php.net/manual/en/class.streamwrapper.php you could do it pretty quickly as the DOMDocument class is loading it (e.g. new DOMDocument('xmlFix:feed.xml')) the worst case scan is when you have a < in the text. Otherwise it's O(N). – Neel Feb 17 '10 at 17:35

score 0 · Answer 3 · answered Sep 17 '15 at 21:46

0

If tidy extension is not an option, you may consider htmlpurifier.

answered Sep 17 '15 at 21:46

lubosdz

4,210
2
29
43

Fix malformed XML in PHP before processing using DOMDocument functions

3 Answers3

Linked

Related