libcurl : Handle multiple asynchronous requests in C++

Question

I've been working with easy_perform till now and it worked as expected. But due to timeouts and single threaded application, there's latency in running multiple operations. I'm looking at optimizing these calls by converting them into asynchronous with multi_perform interface, Though I am having hard time understanding correct way to make use of it.

From my understanding, Flow looks something like following :

1. Create a easy_handle for call

2. Add this standard easy handle to the multi stack using curl_multi_add_handle

3. curl_multi_perform : This is where it gets tricky.

   1. As I understand it, This call is happening in a loop.
   2. My application is calling this API to read/write whatever there is to read or write right now etc.
   3. If running_handles is changed from the previous call, there is data to read which we should retrieve using curl_multi_info_read

4. Clean up when easy handle is processed

   1. curl_multi_remove_handle
   2. curl_easy_cleanup
   3. curl_multi_cleanup

Q:

1. Does that mean, My application needs to do periodic polling to check if there's data to read?

2. Is there a way to handle this with callbacks? and the callback method should trigger action in my application in asynchronous way.

Refs I've already reviewed :

1. Looking at http://www.godpatterns.com/2011/09/asynchronous-non-blocking-curl-multi.html , It says the same thing. Correct me if I'm wrong.

2. stackoverflow thread and other related : How to do curl_multi_perform() asynchronously in C++?

Answer 1

The prerequisite knowledge needed to understand curl_multi API is Async Sockets.

curl_multi_perform is not a blocking API. As explained in documentation:

When an application has found out there's data available for the multi_handle or a timeout has elapsed, the application should call this function to read/write whatever there is to read or write right now etc. curl_multi_perform returns as soon as the reads/writes are done.

It just needs to be called periodically.

Does that mean, My application needs to do periodic polling to check if there's data to read?

Yes. curl_multi_fdset conveniently extracts the related file descriptors so that you can select on them (select = wait), but you're free to add other descriptors to the same select call, thus interleaving curl work with your own work. Here's an example of how to do it.

Is there a way to handle this with callbacks?

Yes. The transferred data is passed during the curl_multi_perform call into a CURLOPT_WRITEFUNCTION callback. Note: curl_multi_info_read is not for reading data, it's for reading information about the transfer.

  for (/* each transfer */) {
    curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb);
    curl_easy_setopt(eh, CURLOPT_WRITEDATA, /* pass userp value */);
    curl_multi_add_handle(multi_handle, easy_handle);
  }
  int still_running;
  do {
    if (curl_multi_perform(cm, &still_running)) { // will call write_cb() when data is read
      /* handle error */ break;
    }
    if (curl_multi_wait(cm, NULL, 0, 1000, NULL)) {
      /* handle error */ break;
    }
  } while(still_running);

Here's a complete example of using a data callback with multi-transfer: 10-at-a-time.
Note: curl_multi_wait used in this example is a convenience wrapper around a select call.