PHP while loop to process faster , maybe with PHP worker implementation

Question

I would like to get idea on how to process a while loop which consist of more than 3000 rows

while ($row = mysql_fetch_array($result)) {
    if $row["Status"] == 1 {
        //Curl 3PP and Update DB 
    } else {
        //Curl 3PP and Update DB
    }
}

Since the row of the result is huge and the while loop is taking around 1 hour to complete process it. Is there any suggestion to expedite the process?

I was thinking of having another PHP to spawn workers to process the CURL and UpdateDB part. Something like this :

while ($row = mysql_fetch_array($result)) {
    if ($row["Status"] == 1) {
        exec("php curl_update.php?database=success");
    } else {
        exec("php curl_update.php?database=fail");
    }
}

Is it a good idea or is there any other ways to fulfill the following requirement using PHP or any other library?

In the curl command, i'll be sending transactionID to a 3rd party API to get the status whether it's fail or success. Latency is around 1-5 seconds

What you should do is switch over to PDO and learn foreach loops. — Thrallix, Jul 21 '20 at 12:00
It is likely the _CURL_ that is taking so much time. Can you give us more info about what you're trying to achieve? Also, `=` probably should be `==`. — waterloomatt, Jul 21 '20 at 12:00
@AaronNoHuanKnows, PDO foreach loop . I'm looking it up on google . TQ for some idea — Jeeva Suriyaa, Jul 21 '20 at 12:02
@waterloomatt, the curl will take sometimes 1-5 seconds to return the value from 3PP . The scenario is : I have 2000 rows , i need to curl with some 3rd party system and update to my table accordingly whether the transaction is fail or success. That's the whole concept. Now it's taking quite some time to process and i'm having backlog. So i'm trying to expedite the process to reduce the backlog — Jeeva Suriyaa, Jul 21 '20 at 12:03
Do you need the result the 3rd party is sending back? Do you have a limited amount of calls u can send to them? — DarkBee, Jul 21 '20 at 12:07
`PDO` and `foreach` will not speed things up at all. It is the fact that you're sending out 2000 http requests via CURL. Can you break the CURL request down for us a bit more? What are you sending? Do you really need to send it for each row or can you send a single request? — waterloomatt, Jul 21 '20 at 12:07
@DarkBee, yes the result from the 3rd party is needed. The TPS ( transaction per second ) is around 300-400tps that they're currently accepting. — Jeeva Suriyaa, Jul 21 '20 at 12:09
@waterloomatt, I'm actually sending them the transactionID that customer transacted to through my channel. So i'm curl-ing their API passing in the transactionID as the request param only. I need to them send each row to get the status . — Jeeva Suriyaa, Jul 21 '20 at 12:11
"the curl will take sometimes 1-5 seconds" + "consist of more than 3000 rows" = "taking around 1 hour to process". The math checks out. You can't make this run faster. Assuming this is some kind of reporting script (and not a webpage) then you should reduce the amount of things you're processing in one execution of the script if you can. — Niet the Dark Absol, Jul 21 '20 at 12:14
Well you could try to execute the curl on another process. You need to start the `exec` [async](https://stackoverflow.com/questions/222414/asynchronous-shell-exec-in-php) though and be wary of the TPS, perhaps add a sleep after X records — DarkBee, Jul 21 '20 at 12:15
@waterloomatt, There's no other way mate. That's their limitation. I'm still scratching my head to find other possibilities i can venture to . Creating two script to process it but i scare that i might be processing duplicate rows because there's no indication for me to split it as well from the transactionID. — Jeeva Suriyaa, Jul 21 '20 at 12:17
@NiettheDarkAbsol, it's more of a requery script for status check. If i reduce the amount of row being processed, my backlog will increase more. Now i'm processing less transaction than what i'm receiving every hour — Jeeva Suriyaa, Jul 21 '20 at 12:18
@DarkBee, That's the tricky part and also my plan. So in the while loop i'll pass maybe to PHP B , in the PHP B i'll spawn the worker to exec the curl and update the DB but to limit the worker. For starting maybe 5, but the tricky part is how will the while loop knows that PHP B is ready to accept new request is my challenge — Jeeva Suriyaa, Jul 21 '20 at 12:19
@DarkBee, If the transaction comes in a real-time it shouldn't be any issue. The requery or this current process is for those not in real-time transaction made by customer/client/front-end. I have to cater a process to check those transaction. — Jeeva Suriyaa, Jul 21 '20 at 12:21
No real way around it then, if you process them sequentially. I have never used _curl_multi_init_ but it may help in this scenario. https://stackoverflow.com/a/9311112/296555. I don't know if it'll work for the same URL/domain though. — waterloomatt, Jul 21 '20 at 12:26
@waterloomatt, damn, i'm out of luck. But i'll check the link you've just posted. At least i can have some idea to improve the current process. Thanks mate — Jeeva Suriyaa, Jul 21 '20 at 12:29
@JeevaSuriyaa I wouldn't be bothered wether php b is ready to accept new rquest. Check out this [demo](https://www.darkbee.be/stack/buffer/index.php) of what I mean. *Takes 10 seconds to load the page though* — DarkBee, Jul 21 '20 at 13:00
@DarkBee, is that your site? Thanks man, will check it out and revert. Thanks for the info man — Jeeva Suriyaa, Jul 21 '20 at 13:38
@ÁlvaroGonzález OP's code is not slow, the process is slow cause of the response time of a 3rd party which OP can't control — DarkBee, Jul 21 '20 at 14:54
@DarkBee, That's a nice site , if you can do online compiler more good haha. The hits and monetization on your site will be rewarding — Jeeva Suriyaa, Jul 21 '20 at 16:26

score 1 · Accepted Answer · answered Jul 21 '20 at 13:18

I think you have a few options now including @DarkBee's exec('php ...' > /dev/null 2>/dev/null &'); and also curl_multi_init.

Here's a quick example showing the time difference between initiating 100 requests using curl_multi_init Vs. the traditional curl_exec.

<?php
$time_start = microtime(true);

$nodes = array_fill(0, 100, 'https://www.google.ca');

// Total time: 1.8 seconds
multiHandle($nodes);

// Vs.

// Total time: 13.5 seconds
//traditional($nodes);

$time_end = microtime(true);
$execution_time = ($time_end - $time_start);
echo sprintf('<b>Total Execution Time:</b> %s secs', round($execution_time, 2));

/*****************************************************************/

/**
 * Taken from: https://stackoverflow.com/a/9311112/296555
 */
function multiHandle($nodes)
{
    $node_count = count($nodes);

    $curl_arr = array();
    $master = curl_multi_init();

    for ($i = 0; $i < $node_count; $i++) {
        $url = $nodes[$i];
        $curl_arr[$i] = curl_init($url);
        curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
        curl_multi_add_handle($master, $curl_arr[$i]);
    }

    do {
        curl_multi_exec($master, $running);
    } while ($running > 0);

    for ($i = 0; $i < $node_count; $i++) {
        curl_multi_getcontent($curl_arr[$i]);
    }
}

/**
 * Taken from: https://www.php.net/manual/en/curl.examples.php#88055
 */
function traditional($nodes)
{
    foreach ($nodes as $destination) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $destination);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_exec($ch);
        curl_close($ch);
    }
}

Thank you very much @waterloomatt, Will look into it – Jeeva Suriyaa Jul 21 '20 at 13:41 — Jeeva Suriyaa, Jul 21 '20 at 13:41
@JeevaSuriyaa Just wondering for an update – DarkBee Aug 11 '20 at 08:05 — DarkBee, Aug 11 '20 at 08:05

PHP while loop to process faster , maybe with PHP worker implementation

1 Answers1