5

I need to execute my Python scripts in parallel so I use the following batch file for it:

start python C:\myfolder\1.py
start python C:\myfolder\2.py
start python C:\myfolder\3.py

It works fine, but now I need to run three more scripts in parallel AFTER the above first three finish. How can I specify it in the same batch file?

aschipfl
  • 33,626
  • 12
  • 54
  • 99
sprogissd
  • 2,755
  • 5
  • 24
  • 45
  • I'm not sure it's a great dupe target (since your requirements are slightly different), but have you seen [this question](https://stackoverflow.com/questions/5375609/wait-for-executable-to-finish-running)? – jedwards Jun 19 '18 at 18:02
  • 1
    might be a dupe of [this](https://stackoverflow.com/questions/12577442/waiting-for-parallel-batch-scripts) question – Leo Jun 19 '18 at 18:06
  • 1
    @jedwards If the first three parallel runs are put into one batch file, and the second group is put in a second batch file, then you could create a wrapper batch file that calls both of batch files using the `/wait` syntax mentioned in your link. – Tim Johns Jun 19 '18 at 18:08
  • Do you have `bash` on your version of Windows? If so, it's simple. – Mark Setchell Jun 19 '18 at 18:17
  • Have you tried, either removing `start` from the third line or using the `start /wait` on the third line; before repeating the parallel sequence for your next three lines? – Compo Jun 19 '18 at 18:19
  • @Compo If I add `start /wait`, wouldn't it wait just for the third script to finish before executing the next three scripts? So if the first two scripts are still not finished and the third one is finished, my second set of scripts will be executed. – sprogissd Jun 19 '18 at 18:26
  • Yes, it would only wait for the third to finish, if the first two are still running it would start the second set. I used it as an alternative to @TimJohns suggestion. My suggestion would only be worthwhile should you know which script would take the longest. – Compo Jun 19 '18 at 18:51
  • @Tim Johns Could you please provide an example of the wrapper batch file as a separate answer to this question? It sounds like a good idea. – sprogissd Jun 19 '18 at 19:02
  • Possible duplicate of [Execute multiple batch files concurrently and monitor if their process is completed](https://stackoverflow.com/questions/43754374/execute-multiple-batch-files-concurrently-and-monitor-if-their-process-is-comple) or [How to wait all batch files to finish before exiting?](https://stackoverflow.com/questions/33584587/how-to-wait-all-batch-files-to-finish-before-exiting) – aschipfl Jun 19 '18 at 20:10

2 Answers2

1

You can easily do this in python without using windows batch commands.

You can use the subprocess library to run external scripts simultaneously and then wait for them all to complete.

processes = []
scripts = [
    r'C:\myfolder\1.py',
    r'C:\myfolder\2.py',
    r'C:\myfolder\3.py',
]
for script in scripts:
    p = subprocess.Popen(['python', script])
    processes.append(p)

for p in processes:
    p.wait()

# Run other processes here
Brendan Abel
  • 35,343
  • 14
  • 88
  • 118
  • When I try this, I get a bunch of these errors: `[60360:53644:0619/113817.886:ERROR:page_load_metrics_update_dispatcher.cc(75)] Invalid response_start 0.11 s for parse_start 0.106 s` – sprogissd Jun 19 '18 at 18:43
  • Those are log messages. They're likely being logged by whatever script you're running. – Brendan Abel Jun 19 '18 at 18:44
0

I think it should work like this:

(
    start python C:\myfolder\1.py
    start python C:\myfolder\2.py
    start python C:\myfolder\3.py
) | pause

For an explanation see this answer.

aschipfl
  • 33,626
  • 12
  • 54
  • 99
  • I tried this, and it didn't wait for the first first three scripts to finish. It executed all the scripts below `) | pause` right away. – sprogissd Jun 20 '18 at 00:11
  • Hm, strange... for me it works, execution continues after `pause` only if all Python scripts terminate; of course, I don't know what your Python scripts do; do they write to _STD_OUT_? – aschipfl Jun 20 '18 at 14:33