Asynchronous PHP execution

Question

I tried to run a massive update of field values through an API and I ran into maximum execution time for my PHP script.

I divided my job into smaller tasks to run them asynchronously as smaller jobs...

I found this post and It looks about right but the comments are a little off-putting... Will using curl to run external script files prevent the caller file triggering maximum execution time or will the curl still wait for a response from the server and kill my page?

The question really is: How do you do asynchronous jobs in PHP? Something like Ajax.

EDIT::///

There is a project management tool which has lots of rows of data. I am using this tools API to access the rows of data and display them on my page. The user using my tool will select multiple rows of data with a checkbox, and type a new value into a box. The user will then press an "update row values" button which runs an update script.

this update script divides the hundreds or thousands of items possibly selected into groups of 100.

At this point I was going to use some asynchronous method to contact the project management tool and update all 100 items.

Because when it is updating those items, it could take that server a long time to run its process, I need to make sure that my original page splitting those jobs is no longer waiting for a request from that operation so that I can fire off more requests to update items. and allow my server page to say to my user "Okay, the update is currently happening, it may take a while and we'll send an email once its complete".

    $step = 100;
    $itemCount = GetItemCountByAppId( $appId );
    $loopsRequired = $itemCount / $step;            
    $loopsRequired = ceil( $loopsRequired );

    $process = array();

    for( $a = 0; $a < $loopsRequired; $a++ )
    {
        $items = GetItemsByAppId( $appId, array( 
            "amount" => $step, 
            "offset" => ( $step *  $a ) 
        ) );  

        foreach( $items[ "items" ] as $key => $item )
        {
            foreach( $fieldsGroup as $fieldId => $fieldValues )
            {
                $itemId = $item->__attributes[ "item_id" ];
                /*array_push( $process, array(
                    "itemId" => $itemId,
                    "fieldId" => $fieldId,
                ) );*/
                UpdateFieldValue( $itemId, $fieldId, $fieldValues );
                // This Update function is actually calling the server and I assume it must be waiting for a response... thus my code times out after 30 secs of execution
            }
        }  

        //curl_post_async($url, $params);
    }

Use a queue/worker system! AMQP, ZeroMQ, Gearman etc are your friends. — deceze, May 20 '13 at 10:27
is launching them through command line a viable option for you? — STT LCU, May 20 '13 at 10:27
It's something that needs to be automated and will be of variable lengths. Some jobs may end up taking 5 minutes at a time... If I can divide the job up into smaller amounts and do these jobs async, then I can simply send an email to the user once the operation has finished... So, to be honest... I'm not completely sure if that's a viable solution — Jimmyt1988, May 20 '13 at 10:29
As for the queue system, is there something that doesn't require extra extensions? I have no control over the current server I am on. — Jimmyt1988, May 20 '13 at 10:32
Then you have something of a problem. If you need to run long jobs, a standard shared host Apache-only web server is a bad platform to be locked into. — deceze, May 20 '13 at 10:35

score 0 · Answer 1 · answered May 20 '13 at 11:24

0

If you are using PHP-CLI, try Threads, or fork() for non-thread-safe version.

answered May 20 '13 at 11:24

SlyChan

769
4
15

And this will stop maximum execution occurring if I fire my jobs off in small amounts on new threads? – Jimmyt1988 May 20 '13 at 11:27
2

It won't. You need to set_time_limit(0) so the script can go on indefinitely. However there's a whole mechanism behind this, how you pass the messages, how you construct replies etc, just using a thread or forked process won't do much on its own. You should research about job queues like deceze said, or you'll end up making your own (using threads or forked processes). – N.B. May 20 '13 at 11:31

score 0 · Answer 2 · answered May 20 '13 at 11:41

Depending on how you implement it, asynchronous PHP might be used to decouple the web request from the processing and therefore isolate the web request from any timeout in the procesing (but you could do the same thing within a single thread). Will breaking the task into smaller concurrent parts make it run faster? Probably not - usually this will extend the length of time it takes for the job to complete - about the only time this is not the case is when you've got a very large processing capacity and can distribute the task effective (e.g. map-reduce). Are HTTP calls (curl) an efficient way to distribute work like this? No. There are other methods, including synchronous and asynchronous messaging, batch processing, process forking, threads....each with their own benefits and complications - and we don't know what the problem you are trying to solve is.

So even before we get to your specific questions, this does not look like a good strategy.

Will using curl to run external script files prevent the caller file triggering maximum execution time

It will be constrained by whatever timeouts are configured on the target server - if that's the same server as the invoking script, then it will be the same timeouts.

will the curl still wait for a response from the server and kill my page?

I don't know what you're asking here - it rather implies that there are functional dependenciesyou've not told us about.

It sounds like you've picked a solution and are now trying to make it fit your problem.

Your bottleneck is most likely the database - sharding the work to run in parallele won't make it go any faster. Invoking the heavy listing outside the webserver wil avoid the timeout problem ( http://symcbean.blogspot.co.uk/2010/02/php-and-long-running-processes.html ) but there are better approaches - e.g. update each row as soon as the user interacts with it, restrict the UI so they can't update more than 100 rows at a time (and multiplex the queries). — symcbean, May 20 '13 at 12:17

Asynchronous PHP execution

2 Answers2