0

I was using Guzzle Pool to request data (around 1-2 MB with each request) from an external server for the past 6 months. I'm doing 5 simultaneous requests at once. However, things have changed and the external server seems to be overloaded and therefore became very slow. Sometimes it's very fast like 1-2 seconds, but many times the server needs to wait 2+ minutes for each request.

This shouldn't be an issue though. However nowadays (since it becomes slow to make requests), some of my requests in the pool are returning an error:

cURL error 18: transfer closed with outstanding read data remaining

It usually does that after waiting for exactly 2 minutes.

The interesting thing though is that if I make the request through Postman (for example) then, I still have to wait 2-3+ minutes, but I got the response in the end.

So this led me to believe that Guzzle's blocking the requests after 2 minutes. I, however, couldn't find any settings to change this. I have even tried sending a Keep Alive and a Content-Length header, but they did not work (maybe I didn't use them correctly though).

Here is a part of my current code that's doing the Guzzle Pool requests. (I'm using PHP 7.1, Guzzle 6.3 and Laravel 5.7).

$headers = ['Authorization' => 'Bearer ' . $token];

$client  = new Client();

$requests = function ($urls, $headers)
{
    foreach ($urls as $key => $url)
    {
        yield new Requests('GET', $url, $headers);
    }
};

$pool = new Pool($client, $requests($urls, $headers),
[
    'concurrency' => 5,
    'fulfilled'   => function ($response, $index)
    {
        echo 'fulfilled -> ' . $index;
    },
    'rejected' => function ($reason, $index)
    {
        echo 'rejected -> ' . $index . ' -> error:' . $reason->getMessage();
    },
]);

$promise = $pool->promise();
$promise->wait();

Unfortunately, I cannot share the external server URL, because it's private.

What am I doing wrong here, that doesn't allow the request to wait for it to finish / send over the data?

Update: I have tried @Alexey Shokov's recommendation, which got rid of the rejected states. The requests are not timing out at 2 minutes now. However I get null as a response once I get something back from those previously timed out sources.

Radical_Activity
  • 2,618
  • 10
  • 38
  • 70

1 Answers1

1

Looks like there is an issue on the server side. The error indicated that cURL client sees a mismatch between expected response size and the real response size, sent by the server.

Take a look at this SO topic, same issue. I think it worths trying to set HTTP version to 1.0 as stated in the discussion with Guzzle's version option or directly in the Request object (yield new Requests('GET', $url, $headers, null, '1.0')).

Alexey Shokov
  • 4,775
  • 1
  • 21
  • 22
  • Thanks for the answer. I have tried doing this and I don't get any errors (rejects) anymore however I'm getting 'null' in the response data. Why could it be that it's working pretty good on a software like PostMan, but not working with Guzzle? – Radical_Activity Apr 05 '19 at 18:33