Multiprocessing of python scripts

Question

I would like to run multiple python scripts (lets say script1.py to script10.py) in parallel in a bash shell. Can someone advise what is the best way to run all simultaneously using xargs command? or is there any other suggestions? Thanks in advance

Why exactly do you want to run it in a bash shell? If there's no specific reason, I'd suggest you to use multiprocessing package to spawn a process for each script and execute them in parallel. — Vignesh Bayari R., Feb 07 '19 at 07:23
Have a look at this https://stackoverflow.com/a/53912275/2836621 — Mark Setchell, Feb 07 '19 at 07:33

score 2 · Accepted Answer · answered Feb 07 '19 at 07:29

2

With bash:

for i in {1..10}; do 
    python script${i}.py &
done

answered Feb 07 '19 at 07:29

rene-d

343
1
6

and use `wait` bash command if you want to wait for the tasks finish. – rene-d Feb 07 '19 at 07:36
Thanks @rene-d. Once I run a single code as (python script1.py) in a bash command it works but when I save the loop in a bash file and then run, it says: python: command not found. Any idea why that doesn't work? – Soheil Feb 07 '19 at 07:45
What is your environnement, Linux/macOS ? Windows ? Basically in Linux or macOS I can't see no difference between running from prompt or running from a script file the python command (unless python is a bash alias). – rene-d Feb 07 '19 at 08:02
I'm using Windows10, however I installed bash and defined python path in my bashrc. But that doesn't work when I execute bash script – Soheil Feb 07 '19 at 08:05
and if you type bash, got the bash prompt and type python : what happens? (probably command not found) Personnaly I gave up using python on my w10 computer :p – rene-d Feb 07 '19 at 12:24

score 0 · Answer 2 · answered Feb 07 '19 at 07:42

There are some considerations you need to take into account. (that other answers haven't taken into account)

Are the scripts developed to run independently? If that is the case this is easier. If not then you will need a pipeline of scripts to manage the dependencies like Python's Luigi.
The scripts may require different python interpreter versions. (2.7 vs 3.7 etc..)
If you do decide to continue in Bash then the usage of nohup would make sure the process doesn't quit when the calling process is ended.

This isn't as easy of a question as you think it might be.

Sorry for not being clear. All scripts are independent. – Soheil Feb 07 '19 at 07:47 — Soheil, Feb 07 '19 at 07:47

score 0 · Answer 3 · answered Feb 07 '19 at 11:18

0

I'm a fan of xargs -P<number> for this kind of use cases, specially because it'll wait for all running children to finish.

Say you have several scripts and but still e.g. at most 4 in parallel, you could:

ls -1 ./script*.py | xargs -I% -P4 sh -c '%'

If you happen to need passing more arguments to your scripts, just add them after above % (inside the quotes)

answered Feb 07 '19 at 11:18

jjo

2,595
1
8
16

Thanks @jjo. Once I run the line it gives me the ImportError message. Seems like that xargs can't find specific python path. Is there any way to address python path? I already defined it in my bashrc. – Soheil Feb 08 '19 at 06:20

Multiprocessing of python scripts

3 Answers3