Parallel script execution from PHP and waiting

Question

I have to start many scripts from PHP and finish them as fast as possible. These scripts are written in a different language, though. I’ve started them somehow but I don’t know how to wait for them. My code:

foreach ($commands as $command) {
    exec($command.' &');
}

The PHP script stop execution but there is still a lot of work in the background. How to wait in PHP for these tasks?

Similar questions:
Parallel processing in PHP - How do you do it? – waiting/synchronisation ignored
Executing multiple PHP scripts in parallel, and being notified when finished – limited to PHP scripts

Do you control these scripts and can add additional functionality? And why are you executing background process if you need to wait for it? — Justinas, May 17 '16 at 09:06
Possible duplicate of [*Parallel processing in PHP - How do you do it?*](http://stackoverflow.com/questions/6107339/parallel-processing-in-php-how-do-you-do-it) or [*Executing multiple PHP scripts in parallel, and being notified when finished*](http://stackoverflow.com/questions/11747304/executing-multiple-php-scripts-in-parallel-and-being-notified-when-finished) — h2ooooooo, May 17 '16 at 09:09
@Justinas I control only PHP. I have many scripts to execute, many cores to use and lack of time. — Michas, May 17 '16 at 09:11
@h2ooooooo I don’t execute parallel PHP scripts. I only start them from PHP. — Michas, May 17 '16 at 09:12
@Michas They might only be for PHP scripts (because the only way you can get notified back when running them with `nohup` or any other parallel command is by the program returning in a special way) but you could simply make a PHP script launcher so it goes `MainPHP -> many PHPScriptLaunchers -> exec()`. That way you'd solve the issue only being in PHP. If that isn't a possibility, please read [this solution](http://stackoverflow.com/a/210919/247893). — h2ooooooo, May 17 '16 at 09:25

score 0 · Answer 1 · answered May 25 '16 at 21:03

If the scripts to execute are written in different languages, the best approach is probably GNU parallel:

Note the documentation:

If command is given, GNU parallel solve the same tasks as xargs. If command is not given GNU parallel will behave similar to cat | sh.

So you can easily build, in PHP, a list of task to do separated by newlines, and feed them to parallel like this:

echo $'ls\necho 42\npython3 -c "print(43)"' | parallel 
1
2
3
42
43

Feeding the list of tasks to parallel's stdin is not that hard using proc_open in PHP.

Then you'll just have to wait for parallel to complete, when parallel has terminated, all subtasks are terminated too, and you can easily configure how many tasks are executed in parallel, if you have too many, executing them all in parallel like you're trying to do will be slower than starting them 5 by 5 like parallel can typically do.

Parallel script execution from PHP and waiting

1 Answers1