3

I am trying to get search results from yahoo.com.

But file_get_contents() converts UTF-8 charset (charset, that yahoo uses) content to ISO-8859-1.

Try:

$filename = "http://search.yahoo.com/search;_ylt=A0oG7lpgGp9NTSYAiQBXNyoA?p=naj%C5%A1%C5%A5astnej%C5%A1%C3%AD&fr2=sb-top&fr=yfp-t-701&type_param=&rd=pref";

echo file_get_contents($filename);

Scripts as 

header('Content-Type: text/html; charset=UTF-8');

or

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

or

$er = mb_convert_encoding($filename , 'UTF-8');

or

$s2 = iconv("ISO-8859-1","UTF-8",$filename );

or

echo utf8_encode(file_get_contents($filename));

NOT help, because after getting web content speciall characters as š ť ž are replaced with question marks ???

I would appreciate any kind of help.

vladinko0
  • 33
  • 1
  • 1
  • 4

4 Answers4

8

This seems to be a content negotiation problem as file_get_contents probably sends a request that only accepts ISO 8859-1 as character encoding.

You can create a custom stream context for file_get_contents using stream_context_create that explicitly states that you accept UTF-8:

$opts = array('http' => array('header' => 'Accept-Charset: UTF-8, *;q=0'));
$context = stream_context_create($opts);

$filename = "http://search.yahoo.com/search;_ylt=A0oG7lpgGp9NTSYAiQBXNyoA?p=naj%C5%A1%C5%A5astnej%C5%A1%C3%AD&fr2=sb-top&fr=yfp-t-701&type_param=&rd=pref";
echo file_get_contents($filename, false, $context);
Community
  • 1
  • 1
Gumbo
  • 643,351
  • 109
  • 780
  • 844
3

file_get_contents should not change the charset. The data is pulled in as a binary string.

When checking out the url you provided, this is the header it provides:

Content-Type: text/html; charset=ISO-8859-1

Also, in the body:

<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">

Also, you can't convert UTF-8 losslessly convert to ISO-8859-1 and get the characters back when going back to UTF-8. UTF-8 / unicode supports many many more characters, so the characters are lost in the first step.

In the browser this is not the case, so perhaps you just need to provide a correct Accept-Encoding header to instruct yahoo's system you can accept UTF-8.

Evert
  • 93,428
  • 18
  • 118
  • 189
1

For anyone investigating on this:

The time I spent on encoding issues taught me that rarely php functions "magically" change the encoding of strings. (One of these rare examples is :

exec( $command, $output, $returnVal )

Please note also that the working header set is as follows:

header('Content-Type: text/html; charset=utf-8');

and not:

header('Content-Type: text/html; charset=UTF-8');

As I had a similar issue as the one you describe, it was enough to set the headers properly.

Hope this helps!

Stavros
  • 264
  • 3
  • 12
1
$s2 = iconv("ISO-8859-1","UTF-8//TRANSLIT//IGNORE",$filename );

Better solution...

function curl($url){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_ENCODING, 1);
    return curl_exec($ch);
    curl_close($ch);
}

echo curl($filename);
Dejan Marjanović
  • 19,244
  • 7
  • 52
  • 66
  • Result is: The document has moved here. – vladinko0 Apr 09 '11 at 10:48
  • @vladinko0, I think you need to set `CURLOPT_FOLLOWLOCATION`, I've updated my answer, try again. – Dejan Marjanović Apr 09 '11 at 11:17
  • Now it loads the page, but with the same result as with file_get_contents() it means with question marks. Charset is also converted to ISO-8859-1. – vladinko0 Apr 09 '11 at 11:32
  • It seems that yahoo.com is serving different pages (charsets) depending on your IP (country). I changed your URL to `http://ru.search.yahoo.com` but it doesn't work. Maybe you can achieve something with accept charset headers,refuse ISO-8859-1... – Dejan Marjanović Apr 09 '11 at 11:47