4

In my web site I have some feature to upload data feeds by users. These data feeds normally having more than 30,000 records. And each records need to be go through complex validation process. So simply this feeds processing will take more than 1 hour to complete the task. This long running process unavoidable.

Here this feeds are uploaded by public user. And feed size is less than 5MB text file. So we don't want to give FTP access as well as we don't want to give command line access to those users. Only we prefer to give web access to upload the feeds. And we have to process the feed immediately after they upload. And once processed we have to send them a report of mail also.

Now the problem is the long running webpages throwing some proxy time out issue on many net work (as it take 1 hour to respond). This is the problem of most client net work configuration.

I need a way to trigger a PHP script once the feed uploaded. And that script should run in background at the same time properly complete the page load without delaying the client.

I will make the background to process the data and send mail to the client once the process completed. Cron tab not feasible here as client may use these feature only few times per month and in unexpected time range.

If there any way to trigger a PHP script in background but through web access please let me know.

A sample script to simplify the requirement

Assume this example and help me on modify this script to achieve the result. I have two PHP scripts "thread.php and createfile.php". The file with name "thread.php" will be opened by client through web browser. This script need to execute the script "createfile.php". But it will take 20 seconds to respond. But we want allow it run in background. And show an out put immediately in the browser.

Following are the code on the example scripts.

createfile.php (http://sugunan.net/demo/createfile.php)

<?php
sleep(20);
@unlink("phpfile.txt");
$myfile = fopen("phpfile.txt", "w") or die("Unable to open file!");
$txt = date("h:i:s")."\n";
fwrite($myfile, $txt);
fclose($myfile);
mail("someone@example.com","Background process","Process completed");
?>

created file visible at web url: http://sugunan.net/demo/phpfile.txt

thread.php (http://sugunan.net/demo/thread.php)

<pre>
<?php
echo "Script start at: " . date('h:i:s') . "\n";
include("createfile.php"); //give a method to execute this script but in background without delay
echo "Script end at: " . date('h:i:s');
?>

This will give following out put with the sleep() function delay.

Script start at: 02:53:27

Script end at: 02:53:47

Is there any way to execute the "createfile.php" without include() method. And avoid the 20 second sleep delay while process the "createfile.php" in background?

Please consider I want to avoid the waiting time on client side and want to give the out put on browser immediately.

sugunan
  • 4,408
  • 6
  • 41
  • 66
  • See here: http://stackoverflow.com/a/13872965/379855 – Donal Sep 06 '14 at 13:44
  • 1
    possible duplicate of [PHP threading call to a php function asynchronously](http://stackoverflow.com/questions/13846192/php-threading-call-to-a-php-function-asynchronously) – Marcin Orlowski Sep 06 '14 at 13:59
  • Thanks for your comments and answers. I have place sample codes with accessible web urls. Can you please give a suggestion how can I modify those script to avoid the delay but process in background. – sugunan Sep 06 '14 at 15:09
  • @ Marcin Orlowski: I have tried the parallel processing suggestion from the similar question. It does the parallel processing fine. But this again waiting to give the out put in browser. If that delay happen on browser then client will get the proxy time out issue again. Following example is the one i tried. http://www.mullie.eu/parallel-processing-multi-tasking-php/ – sugunan Sep 06 '14 at 15:31
  • @Marcin Orlowski : I found the working solution from another stack overflow answer: http://stackoverflow.com/questions/3819398/php-exec-command-or-similar-to-not-wait-for-result/3819422#3819422 . The related question and the selected answers are fit in to my requirement. If you want you can make my question as duplicate of that question. – sugunan Sep 06 '14 at 18:03

3 Answers3

1

Use exec()

http://php.net/manual/en/function.exec.php

This will allow you to run the long process in another thread. I've used this to all other PHP, Python or C apps to convert wav to mp3, video conversion, or parsing through large XML files...

Be sure to have it notify you or the user, probably via email, when it completes, and if it was successful.

EDITED with working code at purehuman.info/demo ... it actually writes the file now. This is what I would make thread.php look like:

<?php 
    echo "Script start at: " . date('h:i:s') . "\n"; 
    exec("php -f  createfile.php /dev/null &"); 
    echo "Script end at: " . date('h:i:s'); 
?>

If you are in windows:

<?php 
    echo "Script start at: " . date('h:i:s') . "\n"; 
    exec("start /B php createfile.php"); //obviously change the paths etc where needed..
    echo "Script end at: " . date('h:i:s'); 
?>

need to give credit for the windows part: How to execue PHP scripts in the background using EXEC() and CMD

Community
  • 1
  • 1
Chemdream
  • 618
  • 2
  • 9
  • 25
  • I put your last edited code here: http://sugunan.net/demo/thread1.php still delay is there. Click and see how long it take. – sugunan Sep 06 '14 at 17:55
  • But you do see the code is working on my site, right? You can see the file is updating, and it's taking zero seconds? – Chemdream Sep 06 '14 at 17:57
  • Oh wait, are you in Windows? If so, > /dev/null & won't work. you'd replace it with something like: exec("start /B php createfile.php"); – Chemdream Sep 06 '14 at 18:04
  • I'm using godaddy linux hosting. Thanks for helping me. But any how I got answer from following post: http://stackoverflow.com/questions/3819398/php-exec-command-or-similar-to-not-wait-for-result/3819422#3819422 this is same question and working solution. – sugunan Sep 06 '14 at 18:12
  • I initially put in the nohup code, but it wasn't needed. Would you mind removing the -1 from this? I mean, it works :-) – Chemdream Sep 06 '14 at 18:18
  • simply this one line works for the problem `exec('bash -c "exec nohup php createfile.php > /dev/null 2>&1 &"');` need to execute under `bash` and direct the out put to null object. This is not actually PHP. It is the way of unix to the problem. – sugunan Sep 06 '14 at 18:20
  • It works for me with out using the extended bash code. – Chemdream Sep 06 '14 at 18:22
  • Actually, if you look at the answer below Cristian's, it's almost the same as mine, and working for others. It might depend on your config... – Chemdream Sep 06 '14 at 18:25
0

PHP is not designed for this kind of job. But there are tricks to try doing it.

If your feeds have more than 30,000 records each, you will have timeout problems with PHP FPM. So your script must run with PHP CLI (Check your server parameters for PHP CLI timeout).

You should look at Rabbit MQ and program a cron job each minute to consume messages in the queue. Here an example with PHP http://www.rabbitmq.com/tutorials/tutorial-two-php.html

Hope that will help you

Guillaume Mercey
  • 391
  • 3
  • 14
  • CLI is fine. But there is no reason to run the cron for each minute if it is a monthly few time activity. And we need to process this script immediately after client done the upload. – sugunan Sep 06 '14 at 15:24
0

At last I got a working answer from following post: php exec command (or similar) to not wait for result

So I edit the code as follows to get my requirement working. But I don't know the explanation how it work. I only copy the syntax from that answer and past it here with modification.

<?php 
echo "Script start at: " . date('h:i:s') . "\n"; 
exec('bash -c "exec nohup php createfile.php > /dev/null 2>&1 &"');
echo "Script end at: " . date('h:i:s'); 
?>

Here php createfile.php will be the command to execute the PHP script and continue without waiting to output.

Thanks to Cristian

Community
  • 1
  • 1
sugunan
  • 4,408
  • 6
  • 41
  • 66