How can I run a long background process from a Perl CGI program?

Question

I have a problem with a long running CGI and the timeout error:

Timeout waiting for output from CGI script

The client side is a form programmed in jQuery. The user input some data and receives a message that the analyses has been launch. The user don't expect to see receive more messages except an email with a link when the data has been analyzed. So, at this point, the connection with the client is closed, right?

In the server side, a Perl CGI script gets the data and executes some C programs (using Perl's system) to analyze them. This process can take from a few seconds to hours depending on the inputed data.

Then the same CGI program parses the results and sends an email to the user with a link to the results webpage.

Since for some data, the CGI can be running for hours I am getting the error message.

I am assuming that increasing ScriptTimeout is a bad idea. I am not even sure that mod_cgi is installed.

What can I do to avoid this error?

Server: Apache2 running in Mac OS X.

Answer 1

I disabled STDOUT buffering, I tried spawning background processes with system(), I tried combinations of fork() and exec(), I tried invoking background bash shells with background subshell processes, you name it, but nothing would allow the parent CGI process to print the output until the background process completed.

This is what ultimately worked for me:

print "Content-type: text/plain\n\n";
print "Executing background process...\n";

# fork this process
my $pid = fork();
die "Fork failed: $!" if !defined $pid;

if ($pid == 0) {
 # do this in the child
 open STDIN, "</dev/null";
 open STDOUT, ">/dev/null";
 open STDERR, ">/dev/null";
 system('bash -c \'(sleep 10; touch ./test_file)\'&');
 exit;
}

print "The background task will be finished shortly.\n";

The trick was closing STDERR in the child process (by reopening it to /dev/null). Without doing this, our internal webserver thought the parent process was active until the child process ended. Also, unbuffered auto-flushed output did not work with our webserver so I could not print any kind of interim status messages.

Answer 2

The CGI should not be doing this work itself. It should instead simply gather the user input and finish immediately, and then dispatch a separate program to do the work offline. A common solution is to use a worker queue to store these requests from users, and the separate program listens to this queue and performs work as requested.

Edit: Typically, there would be a daemon running all the time that listens to the queue (e.g. at my $work I have a worker daemon that uses Beanstalk::Client and beanstalkd for its job queue), but if you have jobs being added only infrequently, then a cron job is a good first implementation.

As an alternate solution, you can fork your CGI and call exec in the child to start your worker program:

# there is work to be done; dispatch the worker script in a child process.
my $pid = fork;
exec "/path/to/worker/script.pl", $arg1, $arg2 if not $pid;

# parent CGI is still alive; return an acknowledgement to the user and return.

Answer 3

Just to add for the case of a cgi shell script instead of a cgi perl script. Performing a STDOUT buffering disable as suggested in previous answer by redirecting the stdout, stdin and stderr to/from /dev/null on the call to the script that runs in background:

#!/bin/sh

./background.sh </dev/null >/dev/null 2> /dev/null

background.sh invokes the execution of a script or program in background. For example.

#background.sh
# wait one minute and then log date/time in log.dat file
sleep 60; echo date >> log.dat &

In this case the calling scrip returns immediately to the calling web server and web page while the script that waits for one minute is till waiting for the time wait to finish. At the end the date is appended to the log.dat file.

Answer 4

In these situations, I call an external program whose only job is to fork off the process. The first process can dissociate from its child and return immediately while the real work keeps going.

You might also see How can I fork a Perl CGI program to hive off long-running tasks?