GuzzleHTTP Encoding Problems

Question

im trying to use GuzzleHTTP 6 for web scraping, and so far, im not able to find a solution for messedup encoding in response body.

Lets say i want to parse the web page, which returs data in many different languages.

Client Initialization

public function __construct() {
    $this->dataClient = new Client(['base_uri' => 'http://somewebsite.org/{language_code}']);
}

Using Data Client

$request = $this->dataClient->get('/endpoint/' . $data_query . '/');
$response = $request->getBody()->__toString();
$decoded = json_decode($response, true);
foreach ($decoded as $index => $data) {
    $decoded[$index] = str_replace(['<option', '>', '</option>'], '', $data);
}
return $decoded;

Problems:

1. If text is in English, response looks almost fine, except that some of the characters are messed up

   manipulation, thereâ;€™s

Instead of

manipulation, there's

2. If i'm trying to get data in any other languages, thats what i get (data in Russian)

   Ð;Ð;°; Ð;¿;Ð;µ;Ñ;€Ð;²;Ñ;‹Ð;¹; Ð;²;Ð

Instead of

На первый взгляд

And the problem is, if you view website, its all fine and nice, but if you try to scrape it, you will face these problems. So far, im not able to find the source of the problem, neither utf8_decode, or iconv helps me to solve the problem.
Any solutions are highly welcome!

So, here is a small update This is the parsing function:

public function processData($data_query) {
    $request = $this->dataClient->get('/endpoint/' . $data_query . '/');
    $response = $request->getBody()->__toString();
    // echo $response; - Everything is fine, no encoding problems
    // return $response; - Encoding problems
    $decoded = json_decode($response, true);
    // return $decoded; - Encoding problems
    foreach ($decoded as $index => $data) {
        $decoded[$index] = str_replace(['<option', '>', '</option>'], '', $data);
    }
    return $decoded; - Encoding Problems
}

Raw response headers

{
    Date: [
        "Wed, 08 Jun 2016 01:45:30 GMT"
    ],
    Server: [
        "Apache"
    ],
    X-Frame-Options: [
        "SAMEORIGIN"
    ],
    Retry-After: [
        "600"
    ],
    Content-Language: [
        "en-GB"
    ],
    Vary: [
        "Accept-Encoding"
    ],
    Transfer-Encoding: [
        "chunked"
    ],
    Content-Type: [
        "text/html;charset=UTF-8"
    ]
}

Answer 1

I have a similar case (load a XML with Guzzle and parse with SimpleXML), but i know that the source was in ISO-8559-1 but the output from SimpleXML parsed result was scrambled. I tried lots of ways, only this one solved_:

$attribute = mb_convert_encoding((string) $attribute, 'ISO-8859-1', 'UTF-8');

The attribute is some XML node value. I simply convert from the know encoding to the one i want. Hope can help some one...

Answer 2

Have a read of the older SO response posted here: Can Goutte/Guzzle be forced into UTF-8 mode?. Yes it mentions using utf8_decode() but also in conjunction with a fork of Guzzle. Have a look at Guzzle's issue tracker, does it have an issue that sounds similar to yours? If so, comment on it to see if the core dev's will fix it. The above SO post is 3 years old, I'd be surprised if hacks and forks were still needed if the issue were so prevalent.

Perhaps it has been fixed and you now need to ensure that the crawled page is itself sending the correct encoding headers. Note that there is an order of precedence with response headers. I believe it's the webserver that is most important, then the DOM itself, if those headers are not sent or omitted from the webserver's response. But please check this as I'm not 100%.