3

I have a python script that spawns a new Process using multiprocessing.Process class. This process is supposed to run forever to monitor stuff. On Unix I can now use os.kill() to send a signal to that specific process and signal.signal(...) within that process to implement my specific interrupt handler. On Windows things don't work.

I read how to do it using popen. Can I specify the CREATE_NEW_PROCESS_GROUP flag for the Process class also? and How?

here is my example code:

import multiprocessing as mp
import time
import signal
import os
import platform

def my_h(signal, frame):
    print("recieved signal", signal)
    raise InterruptedError

def a_task():
    signal.signal(signal.SIGINT, my_h)
    print("this is 'a_task'", os.getpid())
    try:
        while True:
            print(time.time())
            time.sleep(1)
    except Exception as e:
        print(type(e), e)
    print("'a_task' is at end")


if __name__ == '__main__':
    p = mp.Process(target=a_task)
    p.start()
    time.sleep(1)

    if platform.system() == 'Windows':
        print("send CTRL_C_EVENT")
        os.kill(p.pid, signal.CTRL_C_EVENT)
    elif platform.system() == 'Linux':
        print("send SIGINT")
        os.kill(p.pid, signal.SIGINT)

    time.sleep(3)
    try:
        os.kill(p.pid, signal.SIGTERM)
    except:
        pass
Community
  • 1
  • 1
cima
  • 63
  • 1
  • 5
  • maybe this: http://tweaks.com/windows/39559/kill-processes-from-command-prompt/ – Mohammad Athar Oct 26 '16 at 15:50
  • I think this other solution will fix your issue. http://stackoverflow.com/questions/1230669/subprocess-deleting-child-processes-in-windows#4229404 – Alberto Oct 26 '16 at 15:59
  • ẁell it is not simply about terminating the process but rather ensuring that the `except` clause is executed before stopping – cima Oct 26 '16 at 20:20
  • multiprocessing is self-contained and doesn't provide much in the way of hooks for you to modify how it behaves. The `_winapi.CreateProcess` call is hard coded. I don't recommend monkey-patching it -- especially for something as dubious as this. Windows doesn't implement signals. `CTRL_C_EVENT` and `CTRL_BREAK_EVENT` are console control events, so this only works when running via python.exe or a similar console-mode executable that attaches to an instance of conhost.exe. It can't work via pythonw.exe, another GUI script wrapper, or as a service (service executables are run detached). – Eryk Sun Oct 26 '16 at 23:41

1 Answers1

1

I found a workaround, sorta implementing signaling using multiprocessing.Event class.

The clue was then to find interrupt_main() method (which is in either thread (Python2) or _thread (Python3)) which raises KeybordInterrupt in the main thread, which is the process I want to interrupt.

import multiprocessing as mp
import time
import signal
import os
import threading
import _thread

def interrupt_handler(interrupt_event):
    print("before wait")
    interrupt_event.wait()
    print("after wait")
    _thread.interrupt_main()

def a_task(interrupt_event, *args):
    task = threading.Thread(target=interrupt_handler, args=(interrupt_event,))
    task.start()

    print("this is 'a_task'", os.getpid())
    try:
        while True:
            print(time.time())
            time.sleep(1)
    except KeyboardInterrupt:
        print("got KeyboardInterrupt")
    print("'a_task' is at end")


if __name__ == '__main__':
    interrupt_event = mp.Event()
    p = mp.Process(target=a_task, args = (interrupt_event, tuple()))
    p.start()
    time.sleep(2)
    print("set interrupt_event")
    interrupt_event.set()

    time.sleep(3)
    try:
        os.kill(p.pid, signal.SIGTERM)
    except:
        pass
cima
  • 63
  • 1
  • 5
  • This works, but typically the main thread in each process would wait on or periodically poll the event and set some signal for worker threads to exit gracefully, which can be as simple as a global boolean. – Eryk Sun Oct 27 '16 at 10:09
  • Well, that was my first idea also. But think of a task that is really crunching numbers for quite some time. If I want to interrupt that process, that would mean interrupting the worker (non main) thread. I did not find a way when the worker is not responsive. On Unix I send SIGINT and catch it with try except. The above solution is somewhat similar to that Unix work flow, thats why I used it. But thx anyways for the comments. – cima Oct 27 '16 at 10:35
  • I would expect serious number crunching to be offloaded to a C extension, and in that regard most built-in functions have bad to no support for being interrupted. They release the GIL and go off to do their own thing. If it's the main thread, then you're hosed because Python handles signals on it -- hence the `interrupt_main` function, which sets a flag for the interpreter when it runs on the main thread. – Eryk Sun Oct 27 '16 at 10:46
  • ok maybe the example is misleading as of its simplicity. Image a client application (a python script) that runs on some machine with multiple cores. The main thread of the client now starts multiple processes to do the number crunching using multiple cores. Each subprocess fetches data (from some server application). Now imagine the user decided to quit the client then each subprocess should send a message back to the server that the computation was not successful. So the interrupt of the client should interrupt each subprocess and allow each subprocess to do some cleaning. – cima Oct 27 '16 at 10:55
  • I'm just trying to shed light on how this isn't a panacea. The main thread of a worker process may have released the GIL and is in the middle of some 5-minute number crunching task in a Fortran extension function. You won't be able to interrupt it via `interrupt_main`. That just sets a flag for the interpreter. For a simple example, try this in Python 3 on Linux: `sum(range(2**28))`. You can't interrupt it via Ctrl+C. – Eryk Sun Oct 27 '16 at 11:02
  • the crunching is integrating a set of differential equations using scipy's ode. that is interruptable. – cima Oct 27 '16 at 11:13