Whats the limit on asynchronous request to a PHP script making CURL requests

Question

As the title states: what is the number of asynchronous request that can be made to a PHP script that makes CURL requests and returns some information, or approximate number if hardware dependent.

For example:

A piece of Javascript runs in a loop 50 times making asynchronous get requests via AJAX to the same PHP script, storing the result in an array as they are returned.

How many such requests could reasonably made or can the server only think about one CURL request at a time?

please define time when you say *50 times*, per second, per minute, per hour — dbf, Sep 11 '12 at 22:47
@dbf Just 50 times, but all at once as the requests would be asynchronous. 50 was just an example, could 10 be made simultaneously, or even 2. Are multiple simultaneous requests allowed what I am wondering — Gga, Sep 11 '12 at 22:50
50 times at once? Well, if you are using SESSION, this might lock the PHP file (see [session_write_close](http://php.net/manual/en/ref.session.php) for example). It really depends on Apache I guess, on how much space PHP may acquire before it get's killed. How big would you guess be on what the returning values are? — dbf, Sep 11 '12 at 22:55

Mike · Accepted Answer · 2012-09-12T20:01:13.677

3

What you're looking for are the curl_multi_* functions. An example would be like this:

//create the multiple cURL handle
$mh = curl_multi_init();

// Loop over pages and get set the URL to the cURL queue
foreach ($urls as $url) {

    // Get a cURL handle of the current URL
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    // Set all your other curl_setopt() options here for the transfers


    // Success
    if (gettype($ch) == 'resource' && get_resource_type($ch) == 'curl') {
        curl_multi_add_handle($mh, $ch);
    }
}

// Execute the handles
do {
    $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    if (curl_multi_select($mh) != -1) {
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

I am using that on one of my sites to download around 50 URLs and it works perfectly. Of course, it all depends on the allocated memory for your script. You can also set a timeout so that no URLs can make your script hang for too long.

Edit: To actually answer your question, I'm not sure what the limit of the number of URLs you can input into this function is. I would believe that it varies from server to server depending on connection speed, and memory and possibly CPU if you're doing something processor intensive with the result. But with that said, if all of the requests are being made to the same remote server, you may run into bottleneck issues depending on its configuration. This is independent of whether you are using PHP multiple cURL requests or AJAX. All web servers are made to handle many concurrent requests even to the same script, but you can still run into issues with any of the following situations:

Each of the requests sent are very resource consuming
The remote server needs to lock the database, in which case all requests will be executed one by one, so there would be very little difference in sending them all at once or one at a time.
The remote server uses file-based sessions in PHP and your requests require sessions
The remote server is set up to allow very few maximum concurrent clients (generally Nginx handles many concurrent clients better than Apache since it doesn't need to fork a new process for every request, so if it becomes an issue you might want to look into switching web servers). In this case, all subsequent requests will be queued up to a certain hard limit amount and above that, they will be dropped.

edited Sep 12 '12 at 20:01

answered Sep 11 '12 at 23:03

Mike

23,542
14
76
87

1

Wow, you can just think up something, and PHP has a solution :) – dbf Sep 11 '12 at 23:08
Though I would recommended Mike's answer, this does not make single-threaded PHP run as multi-threaded (on the server side). Each CURL request is takes the time to `connect` + `finish execution queue` + `execute request`. Usually, this is not a problem, assuming your script runs in milliseconds. I've seen a benchmark somewhere that measured this kind of thing... – Christian Sep 11 '12 at 23:51
@Christian, That's not true unless I'm misunderstanding your comment. The connections are made simultaneously. From the PHP doc, they "[Allow] the processing of multiple cURL handles in parallel". I was using regular cURL and when I switched to the multiple cURL handles, the script execution time went from 4 min down to only 10-12 seconds, all else being the same. – Mike Sep 12 '12 at 01:38
@Mike, yup you misunderstood. I said "PHP on the server", that is, the server CURL is connecting to (the other party being the PHP client). While client can do parallel connections, the server's PHP still locks up for each request. – Christian Sep 12 '12 at 08:21
at mike and @Christian, thanks for the info. So it is the case that my server can make simultaneous connections to (say) 10 other servers using multi-CURL, aslong as it is to 10 separate servers? What about doing the same thing but via the AJAX method I described in my question? Would that not be possible as the script on my server that made the single-CURL request would lock each time it was called? – Gga Sep 12 '12 at 08:37
@RodgersandHammertime correct; the limit would be the request handling part on the server, not the client. – Christian Sep 12 '12 at 10:31
@Christian http://stackoverflow.com/questions/1430883/simultaneous-requests-to-php-script This answer seems to suggest otherwise? – Gga Sep 12 '12 at 10:59
@RodgersandHammertime The answer actually reinforces what I said. Depending on the server configuration as well as the possibility of locking, it will limit the number of simultaneous requests. This is in contrast with sending requests from the client, which is usually done without issues. – Christian Sep 12 '12 at 14:32
@Christian sorry, I was a bit confused about the word "server" since both the client and the server are technically servers. I see your point and I will edit my answer accordingly. – Mike Sep 12 '12 at 19:40

Whats the limit on asynchronous request to a PHP script making CURL requests

1 Answers1