How to run commands in parallel in bash?

Question

This has been asked before and there are a lot of answers but my needs are bit more specific:

The processes I need to run are all long-running processes that won't end (normally).
I don't want to just run them on background since I need to see their output.
If any of the processes exits I want the rest to be automatically killed as well.
If I use Ctrl+C or other signal on the main command used to run them, I want to kill all the processes.

For example using & + wait does not satisfy the last point. GNU parallel seems quite complicated and is possibly able to do this but I can't find a good example how to use it.

Look here: https://stackoverflow.com/questions/41926123/kill-background-process-on-sigint .. In short, you have to store the pid of the childs which you create and kill them when your script receives SIGINT — hek2mgl, May 30 '19 at 08:30
How can I run several different commands with GNU parallel? To simplify let's say I need to run `make consume`, `make project` and `make run`. I can't find how to use parallel to do that. Also what the hell is `parallel --halt`? `--halt` is not listed on `parallel --help`... — enumag, May 30 '19 at 08:42

score 3 · Answer 1 · answered May 30 '19 at 13:32

3

After fiddling with GNU parallel for a while this seems to work as I need.

Added the scripts I need to run to my Makefile so that that are available as make consume, make project and make run.

Now I run them in parallel like this:

parallel --ungroup --halt now,success=1,fail=1 make ::: consume project run

With --ungroup I can see the output and killing seems to work as expected too. The only difference from my initial post is that when one of the program ends the others are not killed. But that doesn't really matter.

answered May 30 '19 at 13:32

enumag

830
11
21

You can run make jobs in parallel with `make -j`. – John Kugelman May 30 '19 at 16:08
`--halt now,success=1,fail=1` is not supported. You probably want `--halt now,done=1` – Ole Tange May 30 '19 at 19:37
Also consider using `--linebuffer` instead of `--ungroup`. This way you are guaranteed to get full lines from each job and never see a mix of half-lines. – Ole Tange May 30 '19 at 19:48
Thanks for the tips guys! I'll experiment with these next week. – enumag May 31 '19 at 17:11
1

@OleTange `--halt now,done=1 ` doesn't work: `parallel: Error: --halt now must be followed by ,success or ,fail.` – enumag Jun 03 '19 at 13:37
You need to upgrade >=20170422. – Ole Tange Jun 03 '19 at 19:00

John Kugelman · Answer 2 · 2019-05-30T12:59:58.453

The processes I need to run are all long-running processes that won't end (normally).

I'd make them system services using your OS's init system. Let a system designed to manage long running processes manage your long running processes. Why spend your energy reinventing all the features a good init system already has such as auto-restart, log management, and inter-service dependencies?

Let's assume your init system is systemd, which is what modern Red Hat and Debian distros use.

I don't want to just run them on background since I need to see their output.

systemd automatically captures stdout and stderr and saves them in its journal. Commands like systemctl status <service> and journalctl provide many ways to view and search the log data. You can filter by severity level, or by time, or many other metadata fields. You can look at output for a single service or combined output from multiple services.

If any of the processes exits I want the rest to be automatically killed as well.

systemd lets you express this with BindsTo=, Requires=, or other similar options. See the systemd.unit man page for details.

If I use Ctrl+C or other signal on the main command used to run them, I want to kill all the processes.

You can stop services with systemctl stop <service>. It'll stop the named service and its dependents.

For a sampling of all the things systemd can do for you peruse its manual:

score 0 · Answer 3 · answered May 30 '19 at 12:39

you can use xargs -P0

-P max-procs, --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done. While xargs is running, you can send its process a SIGUSR1 signal to increase the number of commands to run simultaneously, or a SIGUSR2 to decrease the number. You cannot increase it above an implementation-defined limit (which is shown with --show-limits). You cannot decrease it below 1. xargs never terminates its commands; when asked to decrease, it merely waits for more than one ex‐ isting command to terminate before starting another.

While well intended, quoting from the man page without something that is close to a solution of the O.P.'s problem doesn't really help much. Others are likely to downvote. Keep posting but give your experience or solution! Good luck. — shellter, May 30 '19 at 13:49

How to run commands in parallel in bash?

3 Answers3