Problem with calling WINDOWS cmd.exe by using subprocess module: fail to control parallel process limitation

Question

I am now working on Python, writing a code that need to call WINDOWS cmd.exe with subprocess.

I would like to run thousands of processes continuously but only 6 or less processes running at the same time.

Article Control the number of subprocesses using to call external commands in python provides solutions and it seemed work fine on calling programs like Remote Desktop or something, a new program appears after one closed. But I got problem when I apply these codes on calling cmd.exe. Codes are below:

import subprocess
from multiprocessing.pool import ThreadPool as Pool

def worker(cmd): 
    p = subprocess.Popen(cmd, );
    p.wait()

commands = ['C:\Windows\System32\mstsc.exe',
            'C:\Windows\System32\mstsc.exe',
            'C:\Windows\System32\mstsc.exe',
            'C:\Windows\System32\mstsc.exe',
            'C:\Windows\System32\mstsc.exe',]

pool = Pool( processes = 2 );
results =[pool.apply_async(worker, [cmd]) for cmd in commands];
ans = [res.get() for res in results];
This works fine. A new window shows when  one another closed.

This code works fine.

But if the processes were cmd.exe

import subprocess
from multiprocessing.pool import ThreadPool as Pool

def worker(cmd): 
    p = subprocess.Popen(cmd, );
    p.wait()

commands = ['cmd.exe /c start "Test1" /d E:\pyworkspace\FAST\ FAST_RV_W64.exe 334.in',
            'cmd.exe /c start "Test2" /d E:\pyworkspace\FAST\ FAST_RV_W64.exe 893.in',
            'cmd.exe /c start "Test3" /d E:\pyworkspace\FAST\ FAST_RV_W64.exe 9527.in',
            'cmd.exe /c start "Test4" /d E:\pyworkspace\FAST\ FAST_RV_W64.exe 114514.in',
            'cmd.exe /c start "Test5" /d E:\pyworkspace\FAST\ FAST_RV_W64.exe 1919810.in']

pool = Pool( processes = 2 );
results =[pool.apply_async(worker, [cmd]) for cmd in commands];
ans = [res.get() for res in results];

*FAST_RV_64.exe is one of my tool program and *.in is setting file. With calling cmd.exe, all cmd windows jumped out at the same time and the limitation didn't work.

import subprocess
from multiprocessing.pool import ThreadPool as Pool

def worker(cmd): 
    p = subprocess.Popen(cmd, );
    p.wait()

commands = ['cmd.exe /c start "Test1" mstsc.exe',
            'cmd.exe /c start "Test2" mspaint.exe',
            'cmd.exe /c start "Test3" SnippingTool.exe',
            'cmd.exe /c start "Test4" mstsc.exe',
            'cmd.exe /c start "Test5" mspaint.exe']

pool = Pool( processes = 2 );
results =[pool.apply_async(worker, [cmd]) for cmd in commands];
ans = [res.get() for res in results];

If I called program through cmd.exe, it jumps out together, too.

I also tried other method in article above, but no one works.

I wonder why this happened and how can I fix my code.

why are you running through `cmd.exe` rather than just directly? — Sam Mason, Sep 09 '19 at 11:19
@sam-mason Running `cmd.exe` shows the same result as running `FAST_RV_64.exe` directly. `cmd.exe` starts automatically after `FAST_RV_64.exe` executed. — Chunpin, Sep 09 '19 at 11:26

score 0 · Accepted Answer · answered Sep 09 '19 at 11:44

That usage of apply_async() looks a little suspect – how about imap_unordered() instead?

Also, cmd + start doesn't necessarily wait for the subcommand to exit, so that may also be part of your problem.

How about something like this?

import subprocess
from multiprocessing.pool import ThreadPool as Pool


def worker(file):
    p = subprocess.Popen(
        [r"E:\pyworkspace\FAST\FAST_RV_W64.exe", file],
        cwd=r"E:\pyworkspace\FAST",
    )
    p.wait()


files = [
    "334.in",
    "893.in",
    "9527.in",
    "114514.in",
    "1919810.in",
]

with Pool(processes=2) as pool:
    for result in pool.imap_unordered(worker, files):
        pass

My problem can be solved with this code. But I would like to know is there any way to make every process worked in individual window? — Chunpin, Sep 10 '19 at 01:48
It really depends on what FAST_RV_W64 does with its console(s) or window(s). — AKX, Sep 10 '19 at 06:37

score 0 · Answer 2 · answered Sep 09 '19 at 11:44

0

from what you've said, there is no need to run the cmd or start programs here. just remove them from your command lines, they're not doing anything.

you're unnecessarily running two extra programs (cmd, start) when only one is necessary (your program)

if you really want to make your computer do this extra work, you can pass the /wait flag to start, which the docs describe as:

Starts an application and waits for it to end.

but, given what you wrote in your question, you're much better off just running your program directly

answered Sep 09 '19 at 11:44

Sam Mason

15,216
1
41
60

Sorry for not expressing clearly, what I'm trying to express is that applying `cmd.exe` is probably necessary because the tool program is a kind of program with CLI, it open up a `cmd.exe` right after start and begin running codes. So if I can solve the "`cmd.exe` starts together problem", perhaps I can solve my problem with this tool. – Chunpin Sep 10 '19 at 01:38

Problem with calling WINDOWS cmd.exe by using subprocess module: fail to control parallel process limitation

2 Answers2