How do I control the order I receive data when getting a response of a multi-curl in php?

Question

In my scenario I could be required to make over 100 curl requests to get information that I need. There's no way to get this information beforehand, and I don't have access to the server that I will be making the requests to. My plan is to use curl_multi_init(). Each response will come in json. The problem is that I need to receive the information in the order that I placed it otherwise I won't know where everything goes after the response comes back. How do I solve this problem.

Answer 1

Obviously, since the requests are asynchronous, you cannot predict the order in which the responses will arrive. Therefore, in your design, you must provide for each request to include "some random bit of information" – a so-called nonce – which each client will somehow be obliged to return to you verbatim.

Based upon this "nonce," you will then be able to pair each response to the request which originated it – and to discard any random bits of garbage that wander in "out of the blue."

Otherwise, there is no(!) solution to your problem.

Answer 2

When you get the handles back from curl_multi_info_read, you can compare those handles against your keyed list, then of course use the key to know where your response goes. Here's the direct implementation, based on a model I use for a scraper:

// here's our list of URL, in the order we care about
$easy_handles['google']     = curl_init('https://google.com/');
$easy_handles['bing']       = curl_init('https://bing.com/');
$easy_handles['duckduckgo'] = curl_init('https://duckduckgo.com/');

// our responses will be here, keyed same as URL list
$responses = [];

// here's the code to do the multi-request -- it's all boilerplate
$common_options = [ CURLOPT_FOLLOWLOCATION => true, CURLOPT_RETURNTRANSFER => true ];
$multi_handle = curl_multi_init();
foreach ($easy_handles as $easy_handle) {
    curl_setopt_array($easy_handle, $common_options);
    curl_multi_add_handle($multi_handle, $easy_handle);
}
do {
    $status = curl_multi_exec($multi_handle, $runCnt);
    assert(CURLM_OK === $status);
    do {
        $status = curl_multi_select($multi_handle, 2/*seconds timeout*/);
        if (-1 === $status) usleep(10); // reported bug in PHP
    } while (0 === $status);
    while (false !== ($info = curl_multi_info_read($multi_handle))) {
        foreach ($easy_handles as $key => $easy_handle) { // find the response handle
            if ($info['handle'] === $easy_handle) {       // from our list
                if (CURLE_OK === $info['result']) {
                    $responses[$key] = curl_multi_getcontent($info['handle']);
                } else {
                    $responses[$key] = new \RuntimeException(
                        curl_strerror($info['result'])
                    );
                }
            }
        }
    }
} while (0 < $runCnt);

Most of this is boilerplate machinery to do the multi fetch. The lines that target your specific question are:

foreach ($easy_handles as $key => $easy_handle) { // find the response handle
    if ($info['handle'] === $easy_handle) {       // from our list
        if (CURLE_OK === $info['result']) {
            $responses[$key] = curl_multi_getcontent($info['handle']);

Loop over your list comparing the returned handle against each stored handle, then use the corresponding key to fill in your response.