How to parallelize for-loop in bash limiting number of processes

Question

I have a bash script similar to:

NUM_PROCS=$1
NUM_ITERS=$2

for ((i=0; i<$NUM_ITERS; i++)); do
    python foo.py $i arg2 &
done

What's the most straightforward way to limit the number of parallel processes to NUM_PROCS? I'm looking for a solution that doesn't require packages/installations/modules (like GNU Parallel) if possible.

When I tried Charles Duffy's latest approach, I got the following error from bash -x:

+ python run.py args 1
+ python run.py ... 3
+ python run.py ... 4
+ python run.py ... 2
+ read -r line
+ python run.py ... 1
+ read -r line
+ python run.py ... 4
+ read -r line
+ python run.py ... 2
+ read -r line
+ python run.py ... 3
+ read -r line
+ python run.py ... 0
+ read -r line

... continuing with other numbers between 0 and 5, until too many processes were started for the system to handle and the bash script was shut down.

Take a look at: [GNU Parallel](https://www.gnu.org/software/parallel/) — Cyrus, Aug 04 '16 at 18:04
See: [Parallelize Bash Script with maximum number of processes](http://stackoverflow.com/q/38160/3776858) or [Bash: limit the number of concurrent jobs?](http://stackoverflow.com/questions/1537956/bash-limit-the-number-of-concurrent-jobs) — Cyrus, Aug 04 '16 at 18:13
...unfortunately, the accepted answer there (err, as-edited, on the first proposed duplicate) is pretty awful. — Charles Duffy, Aug 04 '16 at 18:14
(btw, `seq` isn't a standardized command -- not part of bash, and not part of POSIX, so there's no reason to believe it'll be present or behave a particular way on any given operating system. And re: case for shell variables, keeping in mind that they share a namespace with environment variables, see fourth paragraph of http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html for POSIX conventions). — Charles Duffy, Aug 04 '16 at 18:42
re: "fails for me", there are lots of reasons it could fail; just posting the code isn't useful, posting a log of stderr when it runs would be much more helpful, inasmuch as it would tell us if you were running it with a non-bash shell, if you were running it with a bash that's too old, etc. — Charles Duffy, Aug 04 '16 at 19:45
(posting a log of stderr with it invoked with `bash -x yourscript` would be even better than *that*, as it would show the actual commands as-invoked). — Charles Duffy, Aug 04 '16 at 19:46
Are you aware that if you are allowed to run your own scripts, then you can do a personal installation of GNU Parallel? Can you elaborate if your reason to avoid GNU Parallel is covered on http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html — Ole Tange, Aug 04 '16 at 19:56
...I'm a bit surprised that, when faced with an answer that explicitly specified "depending on a version of bash new enough to have `wait -n`", running `help wait` and looking for `-n` wasn't one of the first steps. Or just running `wait -n` at a command line and seeing if it gave an error. — Charles Duffy, Aug 04 '16 at 20:19
@tomas, you might want to undelete that so the OP can see it -- they don't have enough reputation to see deleted answers yet. — Charles Duffy, Aug 05 '16 at 02:47
Re: "latest approach" comment -- the latest approach only invokes a fixed number of subprocesses, and can't possibly run more than that number of processes at a time, unless you're doing something like backgrounding the Python code, *which I explicitly told you not to do*. (Or if your Python code self-daemonizes any components). Anyhow, the pattern is perfectly fine, and I can't debug how you're *using* the pattern unless I see your actual implementation. — Charles Duffy, Aug 06 '16 at 20:37
Please show **exactly** how you're trying to apply my answer. You should put the `python run.py` where the stub shows the `echo "Thread $i: Processing $line"`. The `set -x` log does not show that it's being used in this way. — Charles Duffy, Sep 21 '18 at 14:00

score 13 · Answer 1 · answered Aug 04 '16 at 19:26

bash 4.4 will have an interesting new type of parameter expansion that simplifies Charles Duffy's answer.

#!/bin/bash

num_procs=$1
num_iters=$2
num_jobs="\j"  # The prompt escape for number of jobs currently running
for ((i=0; i<num_iters; i++)); do
  while (( ${num_jobs@P} >= num_procs )); do
    wait -n
  done
  python foo.py "$i" arg2 &
done

score 11 · Answer 2 · answered Aug 04 '16 at 22:03

11

GNU, macOS/OSX, FreeBSD and NetBSD can all do this with xargs -P, no bash versions or package installs required. Here's 4 processes at a time:

printf "%s\0" {1..10} | xargs -0 -I @ -P 4 python foo.py @ arg2

answered Aug 04 '16 at 22:03

that other guy

116,971
11
170
194

score 8 · Answer 3 · edited Jan 06 '20 at 14:51

8

As a very simple implementation, depending on a version of bash new enough to have wait -n (to wait until only the next job exits, as opposed to waiting for all jobs):

#!/bin/bash
#      ^^^^ - NOT /bin/sh!

num_procs=$1
num_iters=$2

declare -A pids=( )

for ((i=0; i<num_iters; i++)); do
  while (( ${#pids[@]} >= num_procs )); do
    wait -n
    for pid in "${!pids[@]}"; do
      kill -0 "$pid" &>/dev/null || unset "pids[$pid]"
    done
  done
  python foo.py "$i" arg2 & pids["$!"]=1
done

If running on a shell without wait -n, one can (very inefficiently) replace it with a command such as sleep 0.2, to poll every 1/5th of a second.

Since you're actually reading input from a file, another approach is to start N subprocesses, each of processes only lines where (linenum % N == threadnum):

num_procs=$1
infile=$2
for ((i=0; i<num_procs; i++)); do
  (
    while read -r line; do
      echo "Thread $i: processing $line"
    done < <(awk -v num_procs="$num_procs" -v i="$i" \
                 'NR % num_procs == i { print }' <"$infile")
  ) &
done
wait # wait for all the $num_procs subprocesses to finish

edited Jan 06 '20 at 14:51

Ratio Bound

3
2

answered Aug 04 '16 at 18:12

Charles Duffy

280,126
43
390
441

I tried both your earlier solution and this one. The first solution didn't parallelize at all (ran one process); this one ran all num_iters at once and then crashed the system. – strathallan Aug 04 '16 at 19:22
What's the meaning of `wait -n`? – Aug 04 '16 at 19:33
2

@tomas, `wait -n` waits only for a single process, as opposed for *all* background processes to exit. – Charles Duffy Aug 04 '16 at 19:37
@strathallan, if you'd recorded the stderr, we could look at whether there was (for instance) an error message about your `wait` not supporting `-n`, which is likely. Or any other error messages. I'd strongly suggest running with a much smaller maximum job count (say, a num_procs of 2 and a num_iters of 4) to debug. – Charles Duffy Aug 04 '16 at 19:39
@strathallan, ...also, if you're iterating over contents from a file, a `for` loop is the wrong tool to use in entirety; that's what `while read` loops are for, see http://mywiki.wooledge.org/BashFAQ/001. Also, run your code through http://shellcheck.net/ -- it's got a ton of quoting bugs. – Charles Duffy Aug 04 '16 at 19:41
@strathallan, ...beyond that, I can't say anything useful -- as chepner says, code in a comment is unreadable. Consider https://gist.github.com/ -- good syntax highlighting, no advertising, version control, support for multiple files in a gist (so you can have one file with your code, another with a log of running that code with `bash -x`), etc. – Charles Duffy Aug 04 '16 at 19:43
Shouldn't it be so: `${#curr_jobs[@]} >= num_procs` ? Notice the `=`. – Aug 04 '16 at 20:04
Or just simply `=`? – Aug 04 '16 at 20:06
@tomas, `=` creates failure-cases: If you somehow went over (for instance, by jobs already running before starting the loop), you'd loop off into infinity. I do agree that `>=` is more appropriate. – Charles Duffy Aug 04 '16 at 20:14
It doesn't create failure case only _unveils_ them ! @CharlesDuffy – Aug 04 '16 at 20:19
@tomas, if your idea of "unveiling" a failure case is not to log, not to fail-fast, but to potentially hang a system by creating an unreasonable amount of load... well, I'm glad we're not working together. – Charles Duffy Aug 04 '16 at 20:20
@CharlesDuffy Sorry to disapoint you, but I have a different approach. – Aug 04 '16 at 20:25
@CharlesDuffy And there's a flaw in your logic. Your solution is prone the very failure case you described. Mine would only wait when there are EXACTLY so many jobs running as the set process limit value, while yours may loop as long as the number of "other jobs" is equal OR greater than the set limit. And they might run forever... – Aug 04 '16 at 20:42
@tomas, I'll take a failure case that does nothing forever over a failure case that **according to the OP** "crashed the system". – Charles Duffy Aug 04 '16 at 20:47
Excuse me if this is a basic question, but exactly where in the (linenum % N == threadnum) code would I put the call to execute the python file? – strathallan Aug 04 '16 at 20:50
@strathallan, where the `echo` is -- and *without* any `&` after it, since we're running it in the foreground (of a loop that is itself in the background). – Charles Duffy Aug 04 '16 at 20:51
Solution 2 is not balanced. The modulo series can have bottlenecks. – Aug 05 '16 at 04:04
Certainly true, but we're dealing with a large enough series here that I'd expect the differences in runtime to mostly amortize out. Certainly, though, there'll be some waste at the end (when some processes are finished and others are not). – Charles Duffy Aug 05 '16 at 12:20
This `while` loop command didn't work for me. I changed it to `while ((\`jobs -r | wc -l | tr -d " "\` >= num_procs));` and now it works. what is the purpose of using `read` command and using `-p` argument with `jobs` ? – Amir Masud Zare Bidaki Sep 21 '18 at 08:32
The `while read` loop is reading input from `awk`, which processes your input file -- extracting only the jobs to be run by the individual "thread". Thus, when running this as originally written, you had `num_procs` threads, each with its own `while read` loop and a copy of `awk` extracting a different subset of job for that one thread to run. Changing the loop the way you suggest results in a completely different execution model, unrelated to what this answer was written to accomplish. – Charles Duffy Sep 21 '18 at 13:49
@Amirmasudzarebidaki, ...the OP here was running one job per line from an input file (even though they didn't explain that very clearly in the original question). The answer was thus tailored to that use case. – Charles Duffy Sep 21 '18 at 19:22
@CharlesDuffy I meant the first solution's while: `while read -r -a curr_jobs < <(jobs -p -r) \ && (( ${#curr_jobs[@]} >= num_procs ));` which didn't work for me and forked unlimited simultaneous processes :-? – Amir Masud Zare Bidaki Sep 22 '18 at 07:14
1

Ahh. The advantage of `read -a` to read into an array, and then `${#array[@]}` to test that array's length, is that unlike `wc` or `tr`, it's built into the shell itself -- the code in the first answer requires no external commands, whereas your pipeline has several `mkfifo`/`fork`/`exec` sequences required to execute. I'd have to repro the failure to speak to it. – Charles Duffy Sep 22 '18 at 14:28
1

@Amirmasudzarebidaki, ...that said, I replaced the first answer with an implementation that doesn't depend on process substitutions having access to the parent's job table -- some shell version without that property being the most obvious reason for the first implementation to fail. – Charles Duffy Sep 22 '18 at 14:33

score 3 · Answer 4 · answered Apr 01 '20 at 17:43

3

A relatively simple way to accomplish this with only two additional lines of code. Explanation is inline.

NUM_PROCS=$1
NUM_ITERS=$2

for ((i=0; i<$NUM_ITERS; i++)); do
    python foo.py $i arg2 &
    let 'i>=NUM_PROCS' && wait -n # wait for one process at a time once we've spawned $NUM_PROC workers
done
wait # wait for all remaining workers

answered Apr 01 '20 at 17:43

rtx13

2,580
1
6
22

But, is it possible to abort the command am I executing? I already searched SGNINT approaches but I don't find anything useful which I can apply to this approach, did you achieve it? Thanks. – z3nth10n Nov 09 '21 at 23:39
@z3nth10n that's a more complex question that should be posted separately. – rtx13 Nov 14 '21 at 22:33

Ole Tange · Answer 5 · 2019-05-07T07:44:04.553

Are you aware that if you are allowed to write and run your own scripts, then you can also use GNU Parallel? In essence it is a Perl script in one single file.

From the README:

= Minimal installation =

If you just need parallel and do not have 'make' installed (maybe the system is old or Microsoft Windows):
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
mv parallel sem dir-in-your-$PATH/bin/

seq $2 | parallel -j$1 python foo.py {} arg2

parallel --embed (available since 20180322) even makes it possible to distribute GNU Parallel as part of a shell script (i.e. no extra files needed):

parallel --embed >newscript

Then edit the end of newscript.

score 1 · Answer 6 · answered Aug 06 '20 at 15:09

This isn't the simplest solution, but if your version of bash doesn't have "wait -n" and you don't want to use other programs like parallel, awk etc, here is a solution using while and for loops.

num_iters=10
total_threads=4
iter=1
while [[ "$iter" -lt "$num_iters" ]]; do
    iters_remainder=$(echo "(${num_iters}-${iter})+1" | bc)
    if [[ "$iters_remainder" -lt "$total_threads" ]]; then
        threads=$iters_remainder
    else
        threads=$total_threads
    fi
    for ((t=1; t<="$threads"; t++)); do
        (
            # do stuff
        ) &
        ((++iter))
    done 
    wait
done

How to parallelize for-loop in bash limiting number of processes

6 Answers6

Linked

Related