21

I'm working with Python 2.7 on Windows 8/XP.

I have a program A that runs another program B using the following code:

p = Popen(["B"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
return

B runs a batch script C. C is a long running script and I want B to exit even though C has not finished. I have done it using the following code (in B):

p = Popen(["C"])
return

When I run B, it works as expected. When I run A however, I expected it to exit when B exits. But A waits until C exits even though B has already exitted. Any ideas on what's happening and what possible solutions could be?

Unfortunately, the obvious solution of changing A to look like B is not an option.

Here is a functional sample code to illustrate this issue: https://www.dropbox.com/s/cbplwjpmydogvu2/popen.zip?dl=1

The zip file consists of the following files with the following contents:

A.py

from subprocess import PIPE, Popen
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("\r\n\r\nA: I'll wait for B\r\n")

p = Popen(["C:\\Python27\\python.exe", "B.py"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()

log("A: Done.\r\n")
sys.exit(0)

B.py

from subprocess import Popen, PIPE
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("B: launching C\r\n")

p = Popen(["C.bat"])

log("B: Not waiting for C at all. bye!\r\n")
sys.exit(0)

C.bat

@echo off
echo C: Start long running task : %time% >>  "log.txt"
ping -n 10 127.0.0.1>nul
echo C: Stop long running task : %time% >>  "log.txt"

Any input is much appreciated.

khattam
  • 1,164
  • 1
  • 8
  • 14
  • If I get this right, you have program Am which runs program B which runs program C. Program A _also_ runs program C. Is this correct? – Some programmer dude Nov 06 '12 at 05:27
  • No, program A doesn't run program C directly. It would be great if you took a look at attached example. Thanks. – khattam Nov 06 '12 at 05:32
  • So the second `Popen` is from program B? – Some programmer dude Nov 06 '12 at 05:35
  • Yes. A only runs B and has nothing to do with C directly. – khattam Nov 06 '12 at 15:45
  • Does supplying `close_fds=True` to the second `Popen()` call help? I'm guessing that C inherits the stdout/stderr pipes from A and thus A waits until C closes them. – Simon Nov 06 '12 at 16:48
  • I tried using close_fds=True in second program (B) and it doesn't seem to make a difference. Could you please check the attached sample and see if something can be done? – khattam Nov 06 '12 at 17:30

3 Answers3

28

You could provide start_new_session analog for the C subprocess:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

[1]: Process Creation Flags for CreateProcess()

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • Successfully used this code to create the process group (but without the DETACHED_PROCESS flag) and psutil to kill the process tree (ie. group): http://stackoverflow.com/questions/1230669/subprocess-deleting-child-processes-in-windows – Tomas Aug 22 '16 at 15:31
  • 1
    This only works for me on windows if I had `close_fds=True` to the Windows `kwargs` line: `kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True)` (see [this answer](http://stackoverflow.com/a/13593715/461834)) – jtpereyda Feb 28 '17 at 03:33
  • 1
    @jtpereyda have you noticed that it says that you cannot use `close_fds=True` on Windows if you redirect any of stdin,stdout, stderr? – jfs Feb 28 '17 at 08:14
  • @J.F.Sebastian Yes, I interpreted the [docs](https://docs.python.org/2/library/subprocess.html) to mean that if you use `close_fds=True`, stdin, stdout, and stderr will not work (although the phrasing is a bit ambiguous as to what happens if you actually do use them). Your snippet works very well for me after simply adding `close_fds`, but it would probably be safer to omit stdin/stdout/stderr when the platform is Windows. – jtpereyda Feb 28 '17 at 16:11
  • @jtpereyda: the intent of the code is `std*=DEVNULL` i.e., you can't omit stdin/stdout/stderr here (at least on POSIX -- `close_fds=True` does not close std\* there). You don't need `close_fds=True` unless you have other (non-standard) file descriptors opened (either close them or pass `close_fds=True` -- default on Python 3 on POSIX). On Windows, `close_fds=True` is enough (without redirecting) so that the child process won't inherit parent's file descriptors (I haven't tested what happens if you write to stdout in this case or pass `std*=DEVNULL` *and* `close_fds=True` together on Windows). – jfs Feb 28 '17 at 17:10
  • Combining `DETACHED_PROCESS` and `CREATE_NEW_PROCESS_GROUP` makes no immediate sense. Every process is in a group in Windows, even if it's just the group for the winlogon.exe or wininit.exe session, but the *only* API that uses process groups is `GenerateConsoleCtrlEvent` for sending Ctrl+Break (or Ctrl+C if the process manually enables it, since creating a new group initially disables it) to a group of processes that are attached to one's current console. Injecting the control thread is coordinated between the console instance (conhost.exe) and the Windows session server (csrss.exe). – Eryk Sun Jul 18 '17 at 18:28
  • The equivalent of a Unix process tree in Windows is a Job object. The process should be started suspended (creation flag `CREATE_SUSPENDED`) to ensure it's added to the job before it can spawn other processes that leak out of the job. This works best in Windows 8+ since it supports nested jobs, whereas older versions only allow a single job per process. – Eryk Sun Jul 18 '17 at 18:36
  • 2
    Note that starting from Python 3.7, on Windows it's no longer necessary to set `close_fds=True` - that is now the default and works together with redirecting `stdin`/`stdout`/`stderr`. Details in bug tracker here: https://bugs.python.org/issue19764 – nirvana-msu Jan 06 '20 at 13:02
0

An alternative would be to start C as fully forked-off process in a separate process tree, e.g. via the start command of cmd.exe:

import subprocess
subprocess.Popen(["cmd.exe", "/C", "start notepad"])

Obviously, since it is completely independent, you cannot communicate with it. But you can use psutil to retrieve it's PID to at least monitor or close it, if necessary:

import psutil
for process in psutil.process_iter():
    if process.name() == 'notepad.exe':
        print(process.pid)
Robert
  • 1,357
  • 15
  • 26
-1

Here is a code snippet adapted from Sebastian's answer and this answer:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

I've only tested it personally on Windows.

Community
  • 1
  • 1
jtpereyda
  • 6,987
  • 10
  • 51
  • 80
  • 2
    Note that starting from Python 3.7, on Windows it's no longer necessary to set `close_fds=True` - that is now the default and works together with redirecting `stdin`/`stdout`/`stderr`. Details in bug tracker here: https://bugs.python.org/issue19764 – nirvana-msu Jan 06 '20 at 13:03