"&" in for loop in bash shell

Question

I am using & to run multiple programs together in for-loop in bash shell script

for example,

for ((i=1;i<=45;i=i+1))
do
    
    python pythonfile.py --gpu 4 --seed ${i} &
    python pythonfile.py --gpu 4 --seed ${i} &
    python pythonfile.py --gpu 5 --seed ${i} &
    python pythonfile.py --gpu 5 --seed ${i} &
    python pythonfile.py --gpu 6 --seed ${i} &
    python pythonfile.py --gpu 6 --seed ${i} &
    python pythonfile.py --gpu 3 --seed ${i} &
    python pythonfile.py --gpu 3 --seed ${i} 


done

and I did not put & in that last sentence, because it would be terrible if I run &... 45*8 programs together

but It still runs more programs than 8 programs together.

Why this problem is happening?

Perhaps this a job for GNU parallel? – potong May 13 '21 at 07:31 — potong, May 13 '21 at 07:31

score 4 · Answer 1 · answered May 13 '21 at 07:52

4

This might work for you (GNU parallel):

parallel -j8 python pythonfile.py --gpu {2} --seed ${1} ::: {1..45} ::: 4 4 5 5 6 6 3 3

Runs 8 jobs at a time 360 in total.

answered May 13 '21 at 07:52

potong

55,640
6
51
83

thank you for answering the question! I didn't know this way and it is more effective. Thank you – ylee May 14 '21 at 07:09

score 3 · Answer 2 · answered May 13 '21 at 07:12

You have to wait for the background jobs to finish before starting another iteration:

for ((i=1;i<=45;i=i+1))
do    
    python pythonfile.py --gpu 4 --seed ${i} &
    python pythonfile.py --gpu 4 --seed ${i} &
    python pythonfile.py --gpu 5 --seed ${i} &
    python pythonfile.py --gpu 5 --seed ${i} &
    python pythonfile.py --gpu 6 --seed ${i} &
    python pythonfile.py --gpu 6 --seed ${i} &
    python pythonfile.py --gpu 3 --seed ${i} &
    python pythonfile.py --gpu 3 --seed ${i} &
    wait
done

wait is a shell builtin btw, run help wait in your shell for the help page.

"&" in for loop in bash shell

2 Answers2