1

On my machine, which has eight CPUs, I wish to run a number of Python files. I only want to use 7 CPUs concurrently. Is it possible to automate this process?

Currently, I am doing

pids0=""
for i in $(seq 1.5 0.05 1.8); do
   python3 $i &
   pids0="$pids0 $!"
done
wait $pids0


pids1=""
for i in $(seq 1.85 0.05 2.0); do
   python3 $i &
   pids1="$pids1 $!"
done
wait $pids1

Each loop is designed in way to submit only 7 scripts, thus I have to write several for-loops to perform 100 calculations. As you can see, my approach is incredibly inefficient because I have to wait for 7 submitted python scripts to finish running before I can proceed to next loop.

My goal is to write code that can submit the next task as soon as I have any free CPU available after finishing the previous one without writing several for-loops.

PS: I am using my office computer and I am not an administrator.

Raghvender
  • 63
  • 6

1 Answers1

3

You can use GNU Parallel:

parallel -j 7 python3 script.py ::: $(seq 1.5 0.05 2.0)

EDIT according to Mark Setchell's comment

bugless-soup
  • 71
  • 3
  • 2
    You need to replace `-- arg1 arg2 arg3` with `::: $(seq 1.5 0.05 2.0)` – Mark Setchell Aug 02 '22 at 20:26
  • Thanks @Mark, but as I have said I don't have admin rights, so cannot install gnu parallel. – Raghvender Aug 02 '22 at 20:40
  • 1
    @Raghvender Admin rights aren't needed - see this answer from Ole Tange (the author) https://stackoverflow.com/a/42396008/2836621 – Mark Setchell Aug 02 '22 at 20:42
  • 1
    @Raghvender Someone has already asked a similar question [stackoverflow.com/questions/6441509/how-to-write-a-process-pool-bash-shell](https://stackoverflow.com/questions/6441509/how-to-write-a-process-pool-bash-shell) – bugless-soup Aug 02 '22 at 21:34
  • 2
    While you certainly can do `::: $(seq 1.5 0.05 2.0)` I would personally do: `seq 1.5 0.05 2.0 | parallel -j -1 python3 script.py`. It makes no difference here, but if your next machine has 16 cores, it will use 16-1 of those. It will also work if the input parameters is longer than the longest line (e.g. `seq 1.5 0.0000005 2.0`) – Ole Tange Aug 03 '22 at 08:52