Parallel running multiple python scripts through Multiprocessing

Question

I have multiple independent python scripts which I want to run parallelly. I am trying to achieve this through Multiprocessing library. I tried below code but its keep on running and there is no output.

import os
from multiprocessing import Pool

processes = ['1.py','2.py','3.py']

def run_processes (process):
    os.system('python {}'.format(process))

if __name__ == "__main__":
    pool = Pool (processes=3)
    pool.map(run_processes, processes)

Please advise where am I wrong here.

score 0 · Answer 1 · answered Feb 06 '23 at 14:33

0

You need to use:

if __name__ == "__main__":

like it says in the first paragraph of the documentation on multiprocessing.

answered Feb 06 '23 at 14:33

Mark Setchell

191,897
31
273
432

I tried using this If statement but no luck. Updated the code in the question. – Mohit Garg Feb 06 '23 at 14:52
Why did you make your list of processes into a **tuple** rather than the **list** in the documentation? – Mark Setchell Feb 06 '23 at 15:17
I tried with list as well; but still not working. – Mohit Garg Feb 07 '23 at 13:41
Ok, try changing it to `os.system(f'ls {process}')` If it lists your files, that means you need to put the full path to Python in your command which you can find by running `type python`. If it doesn't list your files, that means your program is not running in the directory you expect. – Mark Setchell Feb 07 '23 at 15:51
I tried this as well but same problem. Code is keep on running and not giving any output. – Mohit Garg Feb 09 '23 at 03:14
Ok, what is the **exact** command you use to run the script please? And can you add a line after `if __name__ ...` with `print(os.getcwd())` – Mark Setchell Feb 09 '23 at 07:28
Sorry Mark, what do you mean by _exact command you use to run the script_ ? – Mohit Garg Feb 09 '23 at 08:31
What did you type in your Terminal to run your script? – Mark Setchell Feb 09 '23 at 08:32
Currently I am using this Jupyter Notebook.. Is it not meant for Jupyter Notebook? – Mohit Garg Feb 09 '23 at 08:36
1

Mmm... maybe try this https://stackoverflow.com/a/54252710/2836621 – Mark Setchell Feb 09 '23 at 08:43

Parallel running multiple python scripts through Multiprocessing

1 Answers1