Guzzle not sending requests concurrently

Question

I am trying to make some concurrent requests using Guzzle. Code:

// Imports
use GuzzleHttp\Pool;
use GuzzleHttp\Client;
use GuzzleHttp\Psr7\Request as PSR7Request;

// Part of code that I have problem with
        $client = new Client([
            'timeout'  => 100
        ]);

        $requests = function ($filters, $carbon_date) {
            foreach ($filters as $filter) {
                yield new PSR7Request('GET', route('filters.run', ['filter' => $filter, 'date' => $carbon_date->format('Y/m/d')]));
                Log::debug("Filter Controller Report -> Filter {$filter} fired");
            }
        };

        $result = collect();
        $pool = new Pool($client, $requests($filters, $carbon_date), [
            'concurrency' => 10,
            'fulfilled' => function ($response) use (&$result) {
                $result = $result->merge(json_decode($response->getBody(), true));
            },
            'rejected' => function ($reason, $index) {
                Log::error("Filter Controller Report -> A filter has failed", compact('reason'));
            }
        ]);

        $pool->promise()->wait();

I am using Laravel, and this code is inside a controller.

Code is being served through NGINX+PHP-FPM.

Each sub request logs its own execution too. The thing is I see something like this in logs:

[2019-10-11 02:10:06] production.DEBUG: Filter Manager -> Filter Filter1 took 23 seconds
[2019-10-11 02:10:06] production.DEBUG: Filter Controller Report -> Filter App\Filter\Filter1 fired
[2019-10-11 02:10:17] production.DEBUG: Filter Manager -> Filter Filter2 took 11 seconds
[2019-10-11 02:10:17] production.DEBUG: Filter Controller Report -> Filter App\Filter\Filter2 fired
[2019-10-11 02:10:34] production.DEBUG: Filter Manager -> Filter Filter3 took 17 seconds
[2019-10-11 02:10:34] production.DEBUG: Filter Controller Report -> Filter App\Filter\Filter3 fired
[2019-10-11 02:10:51] production.DEBUG: Filter Manager -> Filter Filter4 took 17 seconds
[2019-10-11 02:10:51] production.DEBUG: Filter Controller Report -> Filter App\Filter\Filter4 fired
[2019-10-11 02:11:09] production.DEBUG: Filter Manager -> Filter Filter5 took 18 seconds
[2019-10-11 02:11:09] production.DEBUG: Filter Controller Report -> Filter App\Filter\Filter5 fired
[2019-10-11 02:11:09] production.DEBUG: Filter Controller Report -> Filters took 86 seconds

My filters take 10~20 seconds to run, so I expect the whole requests to filters to take around 20 seconds. You can see it takes sum of all times to execute all requests (86 seconds).

What makes me sure that it is not running concurrently, is that the Fire logs appear after actual execution of request, which is not expected.

My result are correct, but the timing is not that good.

So, any help is appreciated!

This doesn't look like how to use Guzzle concurrently, you definitely shouldn't have `yield`. You just need to put all the promises you get from getAsyc function(s) in an array and then call wait on that array. Docs are on their website. — Alex Barker, Oct 11 '19 at 00:22
@AlexBarker It's definitively on their website. See http://docs.guzzlephp.org/en/stable/quickstart.html#making-a-request, last but one example. — vfsoraki, Oct 11 '19 at 00:27

score 2 · Accepted Answer · answered Oct 11 '19 at 00:29

2

Wow, I couldn't find anything about this in Guzzle documentations.

To actually have concurrent requests, you have to install curl extension.

My PHP-FPM (Ubuntu 19.04) did not have it installed. After installing it, everything is working!

Credits to this awesome tiny comment: Guzzle async requests not really async?

answered Oct 11 '19 at 00:29

vfsoraki

2,186
1
20
45

You technically need curl-multi, did composer not complain when you installed guzzle? – Alex Barker Oct 11 '19 at 16:02
@AlexBarker No it did not. And Guzzle documents clearly mention that curl is not required. http://docs.guzzlephp.org/en/stable/overview.html#requirements – vfsoraki Oct 11 '19 at 16:17
I think they should update documents and mention this. – vfsoraki Oct 11 '19 at 16:18
Yah, its not required because they fallback to file_get_contents with less functionality but I thought it spit out warnings about php-curl. Not a huge fan of the way guzzle has been implemented. Glad you figured it out! – Alex Barker Oct 11 '19 at 21:35
I struggled around this for 3 days, never immagine that was some kind of fallback to another http handler if you don't have curl installed. I searched the entire web you saved me. – silvered.dragon Oct 19 '22 at 07:20

Guzzle not sending requests concurrently

1 Answers1