Run multiple python scripts concurrently

Question

How can I run multiple python scripts? At the moment I run one like so python script1.py.

I've tried python script1.py script2.py and that doesn't work: only the first script is run. Also, I've tried using a single file like this;

import script1
import script2

python script1.py
python script2.py

However this doesn't work either.

Do you want to run them at the same time; or du you want to run them one after another? — TidB, Feb 16 '15 at 20:20
Just run 2 instances of Python interpreter. It's OS 'problem', not Python problem. — Łukasz Rogalski, Feb 16 '15 at 20:23
Yeah. What platform? This is a very obvious example of something that should be done with a VERY simple bash script. — Christopher Peterson, Feb 16 '15 at 20:24
what about running each script in their own terminal windows......? — Ulf Gjerdingen, Oct 18 '21 at 10:33

score 101 · Accepted Answer · answered Feb 16 '15 at 20:28

101

With Bash:

python script1.py &
python script2.py &

That's the entire script. It will run the two Python scripts at the same time.

Python could do the same thing itself but it would take a lot more typing and is a bad choice for the problem at hand.

I think it's possible though that you are taking the wrong approach to solving your problem, and I'd like to hear what you're getting at.

answered Feb 16 '15 at 20:28

Christopher Peterson

1,570
1
10
9

7

just wanted to say I appreciate the last line of followup, wish we'd learned more – verybadatthis Mar 06 '16 at 06:51
what does `python script1.py && python script2.py &` do? – marciokoko May 19 '17 at 22:46
2

It does what was requested by the OP - runs two python scripts concurrently. – Christopher Peterson May 21 '17 at 00:55
Is this is the best way to run multiple python scripts ? I mean should i write the python script to run other python script or directly write shell scipts ? Just wanted to know best practice ? – user07 Aug 08 '17 at 13:22
I think that would depend completely on the particular circumstances of the problem you're solving. – Christopher Peterson Aug 09 '17 at 15:07
What if script 1 and script 2 need some user input? How can we modify the Bash? – JOHN Feb 22 '18 at 05:16
Problem with this option is that imports are wrong(e.g. you want to run a python script in a subfolder and have imports in the folder). At least it does not work for me – sqp_125 Apr 30 '20 at 07:03
@windwalker it sounds like you don't have Python installed or at least not in your PATH – Christopher Peterson Jan 14 '21 at 14:38
@Windwalker the one liner version: `python script1.py &; python script2.py &` – Christopher Peterson Jan 14 '21 at 14:39
1

@ChristopherPeterson what is the recommended way to terminate them. I have a programs that run an infinite loop and just need stopping once in a while (once per two days roughly) and I dont have a convienient way to kill 20 scripts. – Matej Novosad Jan 28 '23 at 13:30
What is the difference between using `&` and `&&`? I want to run a cronjob script parrallel with an API script because the cronjob is calling the API. – Azhar Feb 09 '23 at 02:46

score 56 · Answer 2 · answered Feb 16 '15 at 20:25

56

The simplest solution to run two Python processes concurrently is to run them from a bash file, and tell each process to go into the background with the & shell operator.

python script1.py &
python script2.py &

For a more controlled way to run many processes in parallel, look into the Supervisor project, or use the multiprocessing module to orchestrate from inside Python.

answered Feb 16 '15 at 20:25

logc

3,813
1
18
29

19

Probably I posted this before or right after the accepted answer. I could turn the question around and ask why didn't I get any upvotes. – logc Nov 19 '16 at 02:56
1

Interestingly your answer seems to have been posted before the accepted answer. There's a three minute difference. – Noah Nov 19 '16 at 03:57
@logc, +1, in this case the two python instances are completely isolated, correct? If one wanted to compare the output from `script1.py` to the output from `script2.py`, do you know the best method? – Zhubarb Oct 17 '19 at 12:47

score 13 · Answer 3 · answered Aug 25 '21 at 11:55

Adding a wait at the end of the bash script would be advisable, as the script exits when one of the process completes. If we need the script to exit only after all the python process are completed, add a wait at the end of the bash script.

So the script would be

#!/bin/bash
python script1.py ;
python script2.py &
python script3.py &
wait

; at the end of first script is used to run script1 & once finished then start with script2 & script3 in parallel and wait till all of the 3 scripts are completed.

score 11 · Answer 4 · answered Jun 08 '19 at 14:26

11

I had to do this and used subprocess.

import subprocess

subprocess.run("python3 script1.py & python3 script2.py", shell=True)

answered Jun 08 '19 at 14:26

user2757128

289
3
3

5

Are these two scripts running in parallel? – Snedecor Mar 11 '20 at 15:41
@Snedecor, this appears to run in parallel, rather each script is running simultaneously. – Alex Witsil Jan 14 '21 at 21:31
works fine if you are a windows user then just use python instead of using python3 – shailu Mar 31 '21 at 07:30
How can I specify the file locations for the scripts?. – Apr 25 '22 at 08:49

score 4 · Answer 5 · answered Mar 16 '18 at 19:38

I do this in node.js (on Windows 10) by opening 2 separate cmd instances and running each program in each instance.

This has the advantage that writing to the console is easily visible for each script.

I see that in python can do the same: 2 shells.

You can run multiple instances of IDLE/Python shell at the same time. So open IDLE and run the server code and then open up IDLE again, which will start a separate instance and then run your client code.

score 1 · Answer 6 · answered Nov 28 '19 at 06:54

1

The most simple way in my opinion would be to use the PyCharm IDE and install the 'multirun' plugin. I tried alot of the solutions here but this one worked for me in the end!

answered Nov 28 '19 at 06:54

Kishan Vedia

31
4

score 1 · Answer 7 · answered Apr 25 '22 at 15:09

1

you can use

import os
os.system("script1.py && script2.py")

and so on you can add ou as many scripts as you want: just separate with &&

This is for Windows OS

answered Apr 25 '22 at 15:09

SAMEER KULKARNI

11
5

or you can also use 'import os os.system("script1.py &") os.system("script2.py &")' – SAMEER KULKARNI Apr 26 '22 at 05:21

score 0 · Answer 8 · answered Feb 24 '18 at 04:16

0

You can use Gnu-Parallel to run commands concurrently, works on Windows, Linux/Unix.

parallel ::: "python script1.py" "python script2.py"

answered Feb 24 '18 at 04:16

Renjith Thankachan

4,178
1
30
47

Where should I write this? – Snedecor Mar 11 '20 at 15:45

score -2 · Answer 9 · answered Feb 24 '18 at 03:55

I am working in Windows 7 with Python IDLE. I have two programs,

# progA
while True:
    m = input('progA is running ')
    print (m)

and

# progB
while True:
    m = input('progB is running ')
    print (m)

I open up IDLE and then open file progA.py. I run the program, and when prompted for input I enter "b" + <Enter> and then "c" + <Enter>

I am looking at this window:

Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
= RESTART: C:\Users\Mike\AppData\Local\Programs\Python\Python36-32\progA.py =
progA is running b
b
progA is running c
c
progA is running

Next, I go back to Windows Start and open up IDLE again, this time opening file progB.py. I run the program, and when prompted for input I enter "x" + <Enter> and then "y" + <Enter>

I am looking at this window:

Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
= RESTART: C:\Users\Mike\AppData\Local\Programs\Python\Python36-32\progB.py =
progB is running x
x
progB is running y
y
progB is running

Now two IDLE Python 3.6.3 Shell programs are running at the same time, one shell running progA while the other one is running progB.

score -3 · Answer 10 · edited Feb 11 '20 at 12:10

You try the following ways to run the multiple python scripts:

  import os
  print "Starting script1"
  os.system("python script1.py arg1 arg2 arg3")
  print "script1 ended"
  print "Starting script2"
  os.system("python script2.py arg1 arg2 arg3")
  print "script2 ended"

Note: The execution of multiple scripts depends purely underlined operating system, and it won't be concurrent, I was new comer in Python when I answered it.

Update: I found a package: https://pypi.org/project/schedule/ Above package can be used to run multiple scripts and function, please check this and maybe on weekend will provide some example too.

i.e:

 import schedule
 import time
 import script1, script2

 def job():
     print("I'm working...")

 schedule.every(10).minutes.do(job)
 schedule.every().hour.do(job)
 schedule.every().day.at("10:30").do(job)
 schedule.every(5).to(10).days.do(job)
 schedule.every().monday.do(job)
 schedule.every().wednesday.at("13:15").do(job)

 while True:
     schedule.run_pending()
     time.sleep(1)

It does not help running them concurrently, how can we run multiple scripts from inside python concurrently? — Mansoor Akram, Dec 10 '16 at 04:03
In Python concurrent execution is big debate... Please read more GIL global interpreter lock. — Snehal Parmar, Dec 10 '16 at 08:47

Run multiple python scripts concurrently

10 Answers10

Linked

Related