Running multiple script in parallel in an endless loop

Question

I have 20 python scripts and i want to run them in parallel in different bash window, that i can do with the below command and run them in backend parallel:-

python testv.py &
python testv1.py &
python testv2.py &
python testv3.py &
python testv4.py &
python testv5.py &
python testv6.py &
python testv7.py &
python testv8.py &
python testv9.py &
python testv10.py &
python testv11.py &
python testv12.py &
python testv13.py &
python testv14.py &
python testv15.py &
python testv16.py &
python testv17.py &
python testv18.py &
python testv19.py &
python testv20.py &

I converted the above in a bash script :- vaa.sh

#!/bin/bash
python testv.py &
python testv1.py &
python testv2.py &
python testv3.py &
python testv4.py &
python testv5.py &
python testv6.py &
python testv7.py &
python testv8.py &
python testv9.py &
python testv10.py &
python testv11.py &
python testv12.py &
python testv15.py &
python testv16.py &
python testv17.py &
python testv18.py &
python testv19.py &
python testv20.py &

I want to run this for 2 3 hours or say forever till the intervention. How can i achieve this.

I tried to add the vaa.sh in a cronjob for 15 minutes but i want to do it in a way so that as soon as the scripts will finish it should again start whether the total time is 15 mint or 20 min.

You can't fool me - that's 21 programs, not 20. So you want all 20 to start and then you wait till all 20 have finished before starting all 20 again? Or you want to keep all running all the time? So if program9 dies it gets restarted immediately regardless of the others? — Mark Setchell, Nov 28 '18 at 19:47
yes you got me right. and sorry for trying to fool you hehe. — , Nov 28 '18 at 20:57

score 0 · Answer 1 · answered Nov 28 '18 at 19:25

0

while true; do
   vaa.sh &;
   #wait till the background process is done it will loop again.
   wait;
done;

answered Nov 28 '18 at 19:25

Reda Meskali

275
3
9

Filip Młynarski · Accepted Answer · 2018-11-28T21:34:40.563

You could do that using multiprocessing

import os
import time
from multiprocessing import Process

def run_program(cmd):
    # Function that processes will run
    os.system(cmd)

# Creating command to run
commands = ['python testv.py']
commands.extend(['python testv{}.py'.format(i) for i in range(1, 21)])

# Amount of times your programs will run
runs = 1

for run in range(runs):
    # Initiating Processes with desired arguments
    running_programs = []
    for command in commands:
        running_programs.append(Process(target=run_program, args=(command,)))
        running_programs[-1].daemon = True

    # Start our processes simultaneously
    for program in running_programs:
        program.start()

    # Wait untill all programs are done
    while any(program.is_alive() for program in running_programs):
        time.sleep(1)

If you want to stop executing your programs after some desired time you could do this like so.

desired_time = 2 * 60 * 60 # 2 Hours into seconds
start_time = time.time()

while True:
    # Initiating Processes with desired arguments
    running_programs = []
    for command in commands:
        running_programs.append(Process(target=run_program, args=(command,)))
        running_programs[-1].daemon = True

    # Start our processes simultaneously
    for program in running_programs:
        program.start()

    # Wait untill all programs are done or time has passed
    while any(program.is_alive() for program in running_programs) and time.time() - start_time < desired_time:
        time.sleep(1)

    # If desired time has passed exit main loop
    if time.time() - start_time > desired_time:
        break

hey really nice solution. so if i wanna run it 1000 times i will do runs = 10000 but what if i wanna run it for one hour or something or 2 hours like that — , Nov 28 '18 at 20:58
Yes, they all will run in parallel and if you set `runs` to 1000 each of your programs will be executed exactly 1000 times. If you want your program to execute for some amount of time you'd need to add condition that would break the loop if certain amount of time have passed. I'll try to implement it to my aswer in hour because I can't right now. — Filip Młynarski, Nov 28 '18 at 21:03
i need one more help... can you help me in getting answer of this question :- https://stackoverflow.com/questions/53561794/iteration-over-a-pandas-df-in-parallel — , Nov 30 '18 at 18:34

Running multiple script in parallel in an endless loop

2 Answers2