developing a script running the same file in parallel in python

Question

I have a python script, it invokes another python file and runs it. However I need the script to run the same file in parallel for multiple times. I will share the code snippet here .This runs the python file once.

output = os.popen('python py_generator_sm20.py' + options)
print output.read()

How do I parallelize it to run multiple times simultaneously?

Possible duplicate of [How to use threading in Python?](https://stackoverflow.com/questions/2846653/how-to-use-threading-in-python) — Jake, Oct 11 '18 at 20:50
`os.popen` returns immediately (before the called command ends) so you can just call it multiple times to start multiple processes in parallel. — Michael Butscher, Oct 11 '18 at 20:58

Hemerson Tacon · Answer 1 · 2018-10-12T02:31:54.743

~~This maybe not full answer you, because of que output part of your code, but could be a start point.~~ Using the multiprocessing module you can create a pool of workers and then with the subprocess module you can call one instance of your script for each worker and check the output:

import multiprocessing as mp
import subprocess as sp

# an example with two runs
commands = ['python test.py', 'python test.py']
# pass the number of threads that will be working
# if the number of threads < len(commands) the exceed 
# will run in sequence when some process terminate
pool = mp.Pool(processes=2)
# execute the script calls
res = pool.map(sp.check_output, commands)
print(*[item.decode() for item in res])
pool.close()

Attention: the return from check_output is a byte string, so you need to convert it back to string

I tested it with the following simple program:

import time

if __name__ == "__main__":

    print("Running an instance at {}".format(time.ctime()))
    time.sleep(2)
    print("Finished at {}".format(time.ctime()))

And that is the output:

Running an instance at Thu Oct 11 23:21:44 2018
Finished at Thu Oct 11 23:21:46 2018
Running an instance at Thu Oct 11 23:21:44 2018
Finished at Thu Oct 11 23:21:46 2018

As you can see they runned at same time.

Thanks for the response. but how do i use sp.call and how do i even use the output part of the code? — Rohan Somanchi, Oct 11 '18 at 22:58
@RohanSomanchi the first answer that I wrote was incomplete. Now I've edited it and added a way to get the output too. Let me know if missed something else. — Hemerson Tacon, Oct 12 '18 at 02:34

score 0 · Answer 2 · answered Oct 11 '18 at 20:56

I think you need this:

from multiprocessing.dummy import Pool as ThreadPool 
pt = ThreadPool(4) 
results = pt.map(pt_function, pt_array)

or maybe this way (if you have many threading scripts):

from Threading_orders import Thread
class First_time(Thread):
    """
    A threading example
    """    
    def __init__(self, a, b):
        """Инициализация потока"""
        Thread.__init__(self)
        self.a = a
        self.b = b
    def run(self):
        MyThread_1(self.a, self.b)
        MyThread_2(self.a, self.b)
        MyThread_3(self.a, self.b)

developing a script running the same file in parallel in python

2 Answers2