1

I'm busy trying to process the following RSS feed: Yahoo Search RSS, using the following code once the data is fetched:

$response = simplexml_load_string($data);

However, 99% of the chinese characters and strings disappear when I interrogate the simple xml object.

I've tried converting the incoming data to utf8 by doing:

$data = iconv("UTF-8", "UTF-8//TRANSLIT", $data);

But this also doesn't help.

Before the data hits simplexml_load_string its 100% fine. But afterwards, its not.

Any ideas?

lordg
  • 520
  • 5
  • 25

3 Answers3

2

What you describe sounds like an encoding issue. Encoding is like a chain, if it get's broken at one part of the processing, the data can be damaged.

When you request the data from the RSS server, you will get the data in a specific character encoding. The first thing you should find out is the encoding of that data.

Data URL: http://tw.blog.search.yahoo.com/rss?ei=UTF-8&p=%E6%95%B8%E4%BD%8D%E6%99%82%E4%BB%A3%20%E9%9B%9C%E8%AA%8C&pvid=QAEnPXeg.ioIuO7iSzUg9wQIc1LBPk3uWh8ABnsa

According to the website headers, the encoding is UTF-8. This is the standard XML encoding.

However if the data is not UTF-8 encoded while the headers are saying so, you need to find out the correct encoding of the data and bring it into UTF-8 before you go on.

Next thing to check is if simplexml_load_string() is able to deal with UTF-8 data.

I do not use simplexml, I use DomDocument. So I can not say if or not. However I can suggest you to use DomDocument instead. It definitely supports UTF-8 for loading and all data it returns is encoded in UTF-8 as well. You should safely assume that simplexml handles UTF-8 properly as well however.

Next part of the chain is your display. You write that your data is broken. How can you say so? How do you interrogate the simplexml object?

Revisiting the Encoding Chain

As written, encoding is like a chain. If one element breaks, the overall result is damaged. To find out where it breaks, each element has to be checked on it's own. The encoding you aim for is UTF-8 here.

  • Input Data: All Checks OK:
    • Check: Does the encoding data seems to be UTF-8? Result: Yes. The input data aquired from the data URL given, does validate the UTF-8 encoding. This could be properly tested with the data provided.
    • Check: Does the raw xml data mark itself as being UTF-8 encoded? Result: Yes. This could be verified within the first bytes that are: <?xml version="1.0" encoding="UTF-8" ?>.
  • Simple XML Data:
    • Check: Does simple_xml support the UTF-8 encoding? Result: Yes.
    • Check: Does simple_xml return values in the UTF-8 encoding? Result: Yes and No. Generally simple_xml support properties containing text that is UTF-8 encoded, however a var_dump() of the simple_xml object instance with the xml data suggests that it does not support CDATA. CDATA is used in the data in question. CDATA elements will get dropped.

At this point this looks like the error you are facing. However you can convert all CDATA elements into text. To do this, you need to specify an option when loading the XML data. The option is a constant called LIBXML_NOCDATA and it will merge CDATA as text nodes.

The following is an example code I used for the tests above and demonstrates how to use the option:

$data_url = 'http://tw.blog.search.yahoo.com/rss?ei=UTF-8&p=%E6%95%B8%E4%BD%8D%E6%99%82%E4%BB%A3%20%E9%9B%9C%E8%AA%8C&pvid=QAEnPXeg.ioIuO7iSzUg9wQIc1LBPk3uWh8ABnsa';
$xml_data = file_get_contents($data_url);

$inspect = 256;
echo "First $inspect bytes out of ", count($xml_data),":\n", wordwrap(substr($xml_data, 0, $inspect)), "\n";
echo "UTF-8 test: ", var_dump(can_be_valid_utf8_statemachine($xml_data)), "\n";

$simple_xml = simplexml_load_string($xml_data, null, LIBXML_NOCDATA);
var_dump($simple_xml);


/**
 * Bitwise check a string if it would validate 
 * as utf-8.
 *
 * @param string $str
 * @return bool
 */
function can_be_valid_utf8_statemachine( $str ) { 
    $length = strlen($str); 
    for ($i=0; $i < $length; $i++) { 
        $c = ord($str[$i]); 
        if ($c < 0x80) $n = 0; # 0bbbbbbb 
        elseif (($c & 0xE0) == 0xC0) $n=1; # 110bbbbb 
        elseif (($c & 0xF0) == 0xE0) $n=2; # 1110bbbb 
        elseif (($c & 0xF8) == 0xF0) $n=3; # 11110bbb 
        elseif (($c & 0xFC) == 0xF8) $n=4; # 111110bb 
        else return false; # Does not match 
        for ($j=0; $j<$n; $j++) { # n bytes matching 10bbbbbb follow ? 
            if ((++$i == $length) || ((ord($str[$i]) & 0xC0) != 0x80)) 
                return false; 
        } 
    } 
    return true; 
}

I assume that this will fix your issue. If not DomDocument is able to handle CDATA elements. As the encoding chain is not further tested, you might still get encoding issues in the further processing of the data, so take care that you keep the encoding up to the output.

hakre
  • 193,403
  • 52
  • 435
  • 836
  • DOM and SimpleXML both use libxml, so any problem you'd have with one, you'll have with the other. – Josh Davis Jun 08 '11 at 22:46
  • Well I think simplexml has more issues. Anyway, I think the problem is that the input data is not UTF-8 even it's announced so. - I edited the answer to make that more clear that simplexml should work with UTF-8. – hakre Jun 08 '11 at 23:40
  • @hakre Thanks for this. To answer your questions. I save the data received to a text file and it appears correct. I save the data received after loading it into simplexml and 99% of Chinese characters are gone. I also try $item->title on one of the feed items and I get a blank result, and not the chinese. SimpleXML supports UTF-8 by default (see http://www.php.net/manual/en/ref.simplexml.php#79258). – lordg Jun 09 '11 at 12:03
  • @lordg: Thanks for the link. In case the XML file does specify a wrong encoding, simple_xml loading it might scratch it. A probable workaround would be to load it with DomDocument first explicitly specifying the endcoding and the converting it into a simple_xml object ([see here](http://www.php.net/function.simplexml-import-dom)) AFAIK it's easier to deal with encodings with DomDocument, but it can be tricky as well. Another solution (and probably simpler) is to correct the wrong information in the XML prior loading it. It's worth to check the XML response data further I assume. – hakre Jun 09 '11 at 12:19
  • @lordg: I think I found the cause of your issue, I updated my answer. Encoding was not the actual root of the problem it seems, but the way you load the document because of CDATA in the XML (which is sort of an encoding as well, but on another level) ;). However going through the encoding chain helped to find the actual cause. – hakre Jun 09 '11 at 12:46
  • @hakre Thank you, This solved the problem. In my process, following the principles you suggested of checking the chain, I got to the correct encoding answers as above, but the CDATA was the actual problem. Thanks Again! – lordg Jun 10 '11 at 12:35
  • Related: [How to keep the Chinese or other foreign language as they are instead of converting them into codes?](http://stackoverflow.com/a/10834989/367456) and [PHP DomDocument failing to handle utf-8 characters (☆)](http://stackoverflow.com/a/11310258/367456) – hakre Jul 13 '13 at 10:41
1

There are a lot of reasons for encoding issues with PHP. I'd check:

Ryan Doherty
  • 38,580
  • 4
  • 56
  • 63
1

I took a look here: Simplexml_load_string() fail to parse error And after doing what it says (

 $data = file_get_contents('http://tw.blog.search.yahoo.com/rss?ei=UTF-8&p=%E6%95%B8%E4%BD%8D%E6%99%82%E4%BB%A3%20%E9%9B%9C%E8%AA%8C&pvid=QAEnPXeg.ioIuO7iSzUg9wQIc1LBPk3uWh8ABnsa');

$data = iconv("GB18030", "utf-8", $data);

$response = simplexml_load_string($data);

) I can see the Chinese characters, but there is a parse error.

Community
  • 1
  • 1
AR.
  • 1,935
  • 1
  • 11
  • 14
  • It's no need to use `iconv()` because this page is already encode in UTF-8. By the way, GB18030 is usually use in Simplified Chinese pages so it's infrequent in use in Taiwan. The usual use charset in Taiwan is "Big5", and more and more sites use UTF-8. – hit1205 Jan 21 '13 at 01:05