Python 2.7 - How to spawn separate Windows consoles for individual subprocess.call() tasks when using threading?

Question

I have explored this problem and undertaken research here on stackoverflow. I have followed suggestions in these threads, but nothing has worked out for me so far (I think I am not understanding something fundamental):

How can I spawn new shells to run python scripts from a base python script?

Opening a Python thread in a new console window

Execute terminal command from python in new terminal window?

I have a script that takes a list of simulations to run (using a separate Windows program), and uses threading to run n number of models in parallel rather than purely sequentially like in a normal Windows batch file. The model normally spits out a bunch of information to a Windows console so that you can monitor its progress. In my case, I want the main Python script to run in one window / console, and then I want each model / simulation to spawn its own window / console so that each simulation can be monitored independently (or indeed paused / stopped / terminated by simply closing the window).

When I run the Python script from within my editor (I've tried both Enthought Canopy and IDLE), the script works exactly as intended, with separate windows being spawned for each simulation. However, if I run the Python script by double-clicking it in Windows explorer or calling it from the Windows command prompt, the individual simulations do not spawn their own windows. They all dump their output straight into the same window causing a confusing mess (and also not allowing me to terminate simulations by closing its individual window).

To resolve my problem, I have tried following all of the suggestions from the above links. This has included all sort of different combinations of using shell=True, using the Windows "start" command, redirecting output pipes, etc. None of those solutions have worked for me.

I feel that I don't understand why my script works as intended from within Canopy / IDLE, but doesn't work when run directly from the command prompt.

My script is as follows:

from time import sleep
import threading
from subprocess import call

def runmodel(arg):
    call(arg)
    sGlobal.release()

if __name__ == '__main__':    

    n = 6 # maximum number of simultaneous runs
    s = 15 # delay between starts in seconds

    simulations = [] # big list of simulations to run in here - each item in list is another list containing the seperate arguments in the command line

    threads = []
    global sGlobal

    sGlobal = threading.Semaphore(n)

    for arg in simulations:
        sGlobal.acquire()
        t = threading.Thread(target=runmodel, args=(arg,))
        threads.append(t)
        t.start()
        sleep(s)

    for t in threads:
        t.join()

For reference, the Windows command line to run a simulation would look like this (there is a big list of these in my actual script - in the Python script each command is passed to subprocess.call as a list not a string):

"<full file path to model executable>" -some -flags -here "<full file path to model control file>"

I would greatly appreciate any input that might help me better understand or resolve my problem here.

EDIT: clarified that each command line is passed to subprocess.call as a list rather than a string.

EDIT 2: I now have working code thanks to J.F. Sebastian's comment below. The key was to use 'cmd.exe /c start "NAME"' at the start of my command line, join everything up as a string rather than a list, and then pass the command line string to subprocess.call() with shell=True. Working code is thus:

from time import sleep
import threading
from subprocess import call

def runmodel(arg):
    call(arg, shell=True) # not the addition of shell=True - this is now required as the command line is being passed as a string rather than a list - I could not get the syntax with quotation marks and everything else to work without using a string and shell=True.
    sGlobal.release()

if __name__ == '__main__':    

    n = 6 # maximum number of simultaneous runs
    s = 15 # delay between starts in seconds

    simulations = ['cmd.exe /c start "NAME" "<full file path to model exe>" -some -flags -here "<full file path to model control file>"'] # big list of simulations to run in here - each item in list is a string that represents the full command line to run a simulation

    threads = []
    global sGlobal

    sGlobal = threading.Semaphore(n)

    for arg in simulations:
        sGlobal.acquire()
        t = threading.Thread(target=runmodel, args=(arg,))
        threads.append(t)
        t.start()
        sleep(s)

    for t in threads:
        t.join()

It looks like you are creating threads and each thread spawns single subprocess. Have you tried to spawn multiple process from main thread? — Maciej Lach, Aug 20 '15 at 05:57
[Other option to check](http://stackoverflow.com/questions/15899798/subprocess-popen-in-different-console) would be to supply command as a list rather than a string. Use `shlex.split(cmd)` for that. — Maciej Lach, Aug 20 '15 at 06:04
Thanks for the reply. For your first comment, I have to use the threads because that was the only way I could successfully introduce a fixed time delay between the start of any two successive model runs (important for reasons of shared files that each model has to access). On your second comment, the command is already passed as a list (otherwise I understand that subprocess.call wouldn't work). Like I said, the script works exactly as intended from with Canopy and IDLE (just not when run standalone). — katamatsu, Aug 20 '15 at 08:59
@MaciejLach: `shlex.split()` is not appropriate on Windows: (1) it uses a different syntax (2) a string is the native interface there. — jfs, Aug 20 '15 at 16:01
here's [tested code that starts multiple consoles](http://stackoverflow.com/a/19797600/4279) — jfs, Aug 20 '15 at 16:04
Thank you very much for that input. That led me down the right path to a working solution. I ended up using 'cmd.exe /c start "NAME"' at the start of my command line, and then joined up everything into a string and used subprocess.call with shell=True. When I tried similar approaches in the past, I could not get the command line syntax to work correctly. Windows was always trying to read my -flags as files, or was not correctly reading my full file paths (even when they had quotation marks). I've edited my original post with the working code. — katamatsu, Aug 20 '15 at 23:19

score 1 · Accepted Answer · answered Aug 24 '15 at 05:09

I now have working code thanks to J.F. Sebastian's comment. The key was to use 'cmd.exe /c start "NAME"' at the start of my command line, join everything up as a string rather than a list, and then pass the command line string to subprocess.call() with shell=True. Working code is thus:

from time import sleep
import threading
from subprocess import call

def runmodel(arg):
    call(arg, shell=True) # not the addition of shell=True - this is now required as the command line is being passed as a string rather than a list - I could not get the syntax with quotation marks and everything else to work without using a string and shell=True.
    sGlobal.release()

if __name__ == '__main__':    

    n = 6 # maximum number of simultaneous runs
    s = 15 # delay between starts in seconds

    simulations = ['cmd.exe /c start "NAME" "<full file path to model exe>" -some -flags -here "<full file path to model control file>"'] # big list of simulations to run in here - each item in list is a string that represents the full command line to run a simulation

    threads = []
    global sGlobal

    sGlobal = threading.Semaphore(n)

    for arg in simulations:
        sGlobal.acquire()
        t = threading.Thread(target=runmodel, args=(arg,))
        threads.append(t)
        t.start()
        sleep(s)

    for t in threads:
        t.join()

Python 2.7 - How to spawn separate Windows consoles for individual subprocess.call() tasks when using threading?

1 Answers1