Running multiple executable parallel with limitation to 5

Question

I am running some larger amount (100) of calculations. Each calculation needs 5 hours to finish and can run only on one core. So in order to make whole process more efficient i need to run 5 (or more, depends on number of cpus) of them at same time. I have 100 folders and in each is one executable. How can I run all 100 executable in the way that only 5 are running at the same time?

I have tried xargs as seen: Parallelize Bash script with maximum number of processes

cpus=5 find . -name ./*/*.exe | xargs --max-args=1 --max-procs=$cpus echo

I can't find the way to run them. Echo only prints paths on screen.

Do your `.exe` files have exec permissions? How do you normally call them? — vintnes, Apr 13 '19 at 17:08

score 1 · Answer 1 · answered Apr 13 '19 at 16:18

1

You can execute the input as the command by specifying a substitution pattern and using that as the command:

$ printf '%s\n' uname pwd | xargs -n 1 -I{} {}
Linux
/data/data/com.termux/files/home

or more easily but less directly by using env as the command, since that will execute its first argument:

printf '%s\n' uname pwd | xargs -n 1 env

In your case, your full command might be:

find . -name '*.exe' | xargs -n 1 -P 5 env

answered Apr 13 '19 at 16:18

that other guy

116,971
11
170
194

1

Could you please be more specific? What kind of output do you get? – vintnes Apr 13 '19 at 17:09

Running multiple executable parallel with limitation to 5

1 Answers1