PHP API acess multiple calls

Question

I am running calls onto PayPal's TransactionSearch API through PHP cURL.

Unfortunately, the API is very slow to respond, sometimes taking anywhere from 30 seconds to more than 5 minutes (depending on the number of records returned from the API) for a single customer.

At the moment, the script is running off a cron job, and looping through each customer one by one. However, if the number of customers scale up, the entire process would take a very long time (few hours) to complete. This is not good enough.

Essentially, I need to run (and process) multiple API calls simultaneously. What's the best way to achieve this?

Answer 1

Since the bottleneck is the remote server, I suggest using curl_multi_exec. You'll be processing a big number of HTTP connections at once and then process their results in one thread.

This is not the fastest solution, which would be to process responses as soon as they're available in multiple threads, but this approach can make processing 50+ times faster without significant changes.

Answer 2

I would suggest looking into Multi Threading your process, Here is a good Stack question that has some great examples of how to accomplish this.

Answer 3

Using PHP , this is an easy way around to make it have multiple processes:

1. Create a file curlWorker.php : This handles the CURL request and tracking the result ( in database for example ). It gets the required paramters through command line as json string which is decoded within the script.

2. Create a main cron file myCron.php which executes the loop and calls curlWorker.php with paramters given in commandline as a json string :

     for Data-to-process { 
       $cmd = "usr/bin/php  path-to-curlWorker.php  '$jsonStringParamters' > 
               /dev/null 2>/dev/null &";        
   
       exec($cmd);
    }    
   
    // example : /usr/bin/php  curlWorker.php  '{ "uid" : "abc123" , "amount" : 20.3 }'
   

3. You should however monitor how many parallel execs you can execute on your server, and mention that variable in your main cron file myCron.php