2

I have read some answers to the question here but none of them work for me. I have different training files and it takes three days till it finishes but I can reduce the time if I run these files at the same time. Besides, I need to run the test files directly when I finish the training for all methods. 

import os
import subprocess,_multiprocessing
import sys
import gc # Garbage Collector
version = ".".join(map(str, sys.version_info[:3]))
if len(version) >3:
   version=version[:-2];
current_dir = os.path.dirname(os.path.realpath(__file__))
# multiprocessing.Process(["python"+version, os.path.join(current_dir, "main_s_train.py"),os.path.join(current_dir, "SC.py")])
gc.collect()
bots = [subprocess.check_call(["python"+version, os.path.join(current_dir, "main_train.py")]),subprocess.check_call(["python"+version, os.path.join(current_dir, "main_s_train.py")]),subprocess.check_call(["python"+version, os.path.join(current_dir, "SC.py")]),subprocess.check_call(["python"+version, os.path.join(current_dir, "MC.py")])]
modules = map(__import__,bots)
import multiprocessing,subprocess
for bot in (bots):
   p = multiprocessing.Process(target=lambda: __import__(bot))
   p.start()

Is there any suggestion to run multiple python scripts at the same time.

I_Al-thamary
  • 3,385
  • 2
  • 24
  • 37

2 Answers2

6

For Ubuntu, there is an answer here that may work. You can run multiple programs through bash. The "&" tells it to run the program in the background.

python program1.py &
python program2.py &

Since it sounds like you are using a remote server that has Ubuntu, I would recommend using tmux instead. It allows you to open up multiple sessions, run a program on each on and keep them running after you have closed your connection. It also allows you to enter back into each session, if you need to enter/read anything from your programs. I found this guide helpful when I had to do something similar a few months ago.

You can run batch files on Ubuntu as well. I'm not as familiar with running batch files on Ubuntu, but something like the following should work for you. You can also add while loops, if statements, etc.. Anything you would normally type into the shell can be put into the batch file to automatically run your programs or navigate directories.

#!/bin/bash
ECHO starting training program
# without the "&", it waits for your training program to finish running
python training_program.py
ECHO training program completed

# Adding the "&" tells the programs to run in the background
# You can also use tmux instead, if you want to navigate the different programs
python program1.py &
python program2.py &
ECHO training programs running in the background

The file should be saved with a ".sh" extension, then make the file executable by running the following through your shell.

chmod +x your_batch_file.sh

If you are using Windows, you could create a batch file that runs all the programs. Here is an example of the file that you can create with your editor of choice:

# If you don't want to see the outputs/print of your training program, 
#     add @ECHO OFF to the start. If you want to see them, remove the @ECHO OFF
@ECHO OFF

# Without "start" before the script to run your training program,
#     the batch file will wait until the training program finishes
python "path\to\your\training_program.py"
ECHO training program completed

# Adding "start" opens it in a new window, and processes the next line
#     without waiting for the program to finish running
start python "path\to\your\program1.py"
ECHO Running program1
start python "path\to\your\program2.py"
ECHO Running program2

# Adding "PAUSE" makes the script wait for you manually type a key to continue,
#     but it is not required. You can add PAUSE anywhere in the script
PAUSE

"start" runs each program in a new window. After you have configured the text, safe the file with a ".bat" extension. Then all you have to do is click on the file to run the batch file, which will open each program in a separate window.

Similarly, you can just run the following from command prompt and it will open them in separate windows as well. 

start python path\to\your\program1.py
start python path\to\your\program2.py

But it sounds like you are performing this more than once, in which case a batch file may be more suitable.

Mark Hebert
  • 919
  • 1
  • 9
  • 15
  • Thank you for your answer, the problem is that I need the program to run the multiple python scripts when it finishes the training directly. – I_Al-thamary Oct 24 '19 at 03:17
  • 1
    That makes sense. You can create/run a similar batch file in Ubuntu that should handle everything for you. I edited the answer, and hope it works for you. – Mark Hebert Oct 24 '19 at 19:21
  • Thank you very much. How to keep the shell open when using `start`? I mean when I used start the files work quickly and I need to follow the code through printing lines but the console closed quickly. – YasserKhalil Apr 17 '22 at 06:12
1

You could try using the IDE known as PyCharm. It has a plugin called 'Multirun', this allows for the user to run several pythons scripts in parallel. Hope this helps!

Kishan Vedia
  • 31
  • 4