How to run several python scripts in one file concurrently?

Question

I want to run several python scripts from the main file and make them work in parallel. I would like to print their outputs in a console if it is possible. It's better to run them in different processes to be able to operate them (terminate, stop, etc).

I tried to use subprocess module but it does not print the scripts outputs and does not run the scripts concurrently. The:

import subprocess
p1 = subprocess.Popen([sys.executable, "first.py"])
p1.communicate()
p2 = subprocess.Popen([sys.executable, "second.py"])
p2.communicate()

does not run both files but only one.

I tried to use module multiprocessing but it does not allow you to execute files. The:

import multiprocessing
def run(file_path):
    exec(open(file_path).read())
p1 = multiprocessing.Process(target=run, args=(file_path,))

is not an option because it executes file in the current file but I want to run this the same way as if this file was run separately.

I thought I can import main function from the executable files and call it in different processes but I do not know how to do it. And I am not sure it is a good solution because the main function is not supposed to be called from another file.

Answer 1

Your issue is Popen.communicate() waits for the process to finish, then returns output to stdout. See this question.

Try using threading to thread each subprocess and run them in parallel.

import subprocess
from threading import Thread

def run_file(filename):
  p1 = subprocess.Popen([sys.executable, filename])
  p1.communicate()

t1 = Thread(target=lambda:run_file("first.py"))
t2 = Thread(target=lambda:run_file("second.py"))

t1.start()
t2.start()

This will not ensure that first.py necessary will run before second.py however, but by the nature of the question I would assume that does not matter.