Running two python code in parallel from two different directory using Multiprocessing

Question

Below is my code for running two python code in parallel using multiprocessing :

defs.py

import os 

def pro(process):      
    #print(process)                                                       
    os.system('python {}'.format(process))

Multiprocessing.py

import os                                                                       
from multiprocessing import Pool
import multiprocessing as mp
import defs
import datetime
import pandas as pd

processes = ('python_code1.py','python_code2.py')

if __name__ == '__main__':                                                                                
    pool = Pool(processes=4)
    start = datetime.datetime.now()
    print('Start:',start)   
    pool.map(defs.pro, processes)
    end = datetime.datetime.now()
    print('End :',end)   
    total = end-start
    print('Total :', end-start)

This code is running perfectly fine. But my requirement is I need to run the python code 'python_code1.py' and 'python_code2.py' from two different directory.

so I made the below changes in Multiprocessing.py:

path1 = r'C:\Users\code1\python_code1.py'
path2 = r'C:\Users\code2\python_code2.py'

processes = (path1,path2)

but this is not working for me.

My Multiprocessing.py and defs.py are kept on path `C:\Users\Multiprocessing\'

Could you share what error you're getting? I tried and the same code on a linux machine and it worked fine — Jp Reddy, May 29 '20 at 09:45
You really should not use `os.system` - there is a whole module in the standard library - [subprocess](https://docs.python.org/3/library/subprocess.html), which lets you have much more control of how you invoke sub-processes, including setting the directory in which the sub-process is run. — Amitai Irron, May 29 '20 at 09:56
@Jp Reddy : no error, I just got the Start,End,Total time(which is just a fraction of millisecond), but my both the codes are saving excel file in the specific location which is not happening. — Aditya sharma, May 29 '20 at 09:57
@Jp Reddy : So where have You kept your `Multiprocessing.py` and `defs.py` code. In my case these two codes are in different folder. — Aditya sharma, May 29 '20 at 10:02
Why don’t you just call the programs in the shell? Why do you need to run them through system calls within Python? — innisfree, May 29 '20 at 10:34

Hamza · Answer 1 · 2020-05-29T10:52:48.763

Well an elegant solution using asyncio. It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem. And you might find syntax easier as I do:

import os
import asyncio
    def background(f):
        def wrapped(*args, **kwargs):
            return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

        return wrapped

    @background
    def pro(process):      
    #print(process)                                                       
    os.system('python {}'.format(process)) 

processes = (r'C:\Users\code1\python_code1.py',r'C:\Users\code2\python_code2.py')
for process in processes:
    pro(process)

Detailed answer on parallelizing for loop. You might find useful.

This seems not working when i pass the `path` in the `processes`. — Aditya sharma, May 29 '20 at 13:22
I have checked and run it with two different scripts and works fine. Maybe you are passing the path wrong. Most likely error is that path should be like C:\Users\\code1\... Try checking path by running script directly via command prompt or opening the location specified. — Hamza, May 29 '20 at 16:17

Running two python code in parallel from two different directory using Multiprocessing

1 Answers1