python 2.7: how to catch keyboard interrupt in program with more than 25 threads

Question

I want to stop my program when the user presses ctrl-C. The following answer suggests catching the KeyboardInterrupt exception.

python: how to terminate a thread when main program ends

Sometimes it works. But in the following example, it stops working after I increase the number of threads from 25 to 30.

import threading, sys, signal, os

stderr_lock = threading.Lock()

def Log(module, msg):
    with stderr_lock:
        sys.stderr.write("%s: %s\n" % (module, msg))

class My_Thread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        Log("Init", "Initing.")
        self.start()
    def run(self):
        try:
            while True:
                Log("Run", "Running.")
        except KeyboardInterrupt:
            os._exit(0)

for i in range(30):
    My_Thread()

# trap ctrl-C in main thread
try:
    while True:
        pass
except KeyboardInterrupt:
    os._exit(0)

This has a very suspiciously similar feel to the following question:

Thread-Safe Signal API in Python 2.7

In that case, I was unable to catch signals after increasing the number of threads beyond 87.

@JohanL Ubuntu14. Out of curiosity, does it still work if you increase the range(30) to range(1000)? — personal_cloud, Sep 12 '17 at 16:21
At 1000 threads I had the same problem as you. See my answer to why and what you could do about it. — JohanL, Sep 13 '17 at 18:23

score 1 · Accepted Answer · answered Sep 13 '17 at 18:22

1

There are actually two different issues with your code that gives this behavior. The first is that your threads should be made into daemon threads, so that they automatically stops when the main thread exits, the second is that your try block does not encapsulate the thread creation and start-up.

When you create a number of threads, the thread creation won't be finished for quite a while (since it is continuously interrupted by the created threads and the GIL prevents them to run in parallel). Therefore, you send your KeyboardInterrupt before being set up to be handled. However, the KeyboardInterrupt will still kill the main thread (with a Traceback), but not the child threads.

Thus, your code works if you modify it as:

import threading, sys, signal, os

stderr_lock = threading.Lock()

def Log(module, msg):
    with stderr_lock:
        sys.stderr.write("%s: %s\n" % (module, msg))

class My_Thread(threading.Thread):
    def __init__(self, value):
        threading.Thread.__init__(self)
        self.value = value
        Log("Init", "Initing %d." % self.value)
        self.daemon = True
        self.start()
    def run(self):
        while True:
            Log("Run", "Running %d." % self.value)

# trap ctrl-C in main thread
try:
    for i in range(1000):
        My_Thread(i)

    while True:
        pass
except KeyboardInterrupt:
    os._exit(0)

Note, that making the threads into daemons is not strictly necessary in the current example, but I would consider that to be good practice for threads that are supposed to end when the main program ends.

answered Sep 13 '17 at 18:22

JohanL

6,671
1
12
26

"When you create a number of threads, the thread creation won't be finished for quite a while (since it is continuously interrupted by the created threads and the GIL prevents them to run in parallel)." Bingo! I never would have thought of that. Wow, in all my experience in other languages, I have never run into one with such a bad scheduler (even on single-processor systems). Great answer. Yes, now it works. Thank you. – personal_cloud Sep 14 '17 at 03:54
Or, if I had just been patient enough to let all the threads get created, my original example would have worked.... but, as I said, I never would have thought of that. – personal_cloud Sep 14 '17 at 03:57
The `daemon=True` is even better. Somehow I thought that wasn't supported in pre-3.0 Python. I was wrong! Thank you for pointing this out. Actually I wasn't the first one to be confused about pre-3.0 daemon threads: https://github.com/dask/distributed/issues/18. So apparently Python 3.0 added `daemon` as an argument to `Thread` and then people started writing code that wasn't doing it the backward-compatible way. – personal_cloud Sep 14 '17 at 04:02
The `daemon=True` has been available since Python 2.6 and before that you could use `setDaemon` instead, so I think the page you are linking to, had another type of issue, but I don't know. – JohanL Sep 14 '17 at 04:18
As far as the GIL is concerned, it is an annoying implementation detail, but it is not only Python that has it. E.g. Ruby has a similar architecture. Threads works for code that releases the GIL (typically `numpy`) and to simplify code that is mostly waiting (typically sockets). For processor bound code that needs to be run faster, consider to use `multiprocessing` instead. As long as the processes don't communicate or share state, that is rather straight-forward. If you need to share data, it is still possible but requires a bit more of work, typically using queues. – JohanL Sep 14 '17 at 04:19
This just in: `daemon=True` is BAD. See https://stackoverflow.com/questions/20596918/python-exception-in-thread-thread-1-most-likely-raised-during-interpreter-shutd. BAD BAD daemon -- can't debug anything. Went back to `signal.pause()` and I can debug things again. Why couldn't they just not frickin break ctrl-C in the first place????? – personal_cloud Sep 15 '17 at 02:54
Wel, yes, if the threads need to be exited properly or in order, or needs to be synchronized when the program ends, then you want to close them in a proper way, and not make them daemons. In your example you did not have any of that, but I guess it was a down-scaled example? How do you use the signals? It might be better to just set a condition in main, that the threads check every once in a while, to see if they should exit. – JohanL Sep 15 '17 at 04:40
I have found that if you put every thread `run()` AND `start()` in a `try..except` and start your very first imported module with `signal.signal()` and your last module ends with `signal.pause()`, then ctrl-C works reliably. Thank you everyone for your help. – personal_cloud Sep 15 '17 at 06:35

score 0 · Answer 2 · answered Sep 12 '17 at 09:11

You may want to read https://stackoverflow.com/a/35430500/1656850, namely:

There are 3 exit functions, in addition to raising SystemExit.

The underlying one is os._exit, which requires 1 int argument, and exits immediately with no cleanup. It's unlikely you'll ever want to touch this one, but it is there.

sys.exit is defined in sysmodule.c and just runs PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively the same as directly raising SystemExit. In fine detail, raising SystemExit is probably faster, since sys.exit requires an LOAD_ATTR and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit produces slightly smaller bytecode (4bytes less), (1 byte extra if you use from sys import exit since sys.exit is expected to return None, so includes an extra POP_TOP).

The last exit function is defined in site.py, and aliased to exit or quit in the REPL. It's actually an instance of the Quitter class (so it can have a custom repr, so is probably the slowest running. Also, it closes sys.stdin prior to raising SystemExit, so it's recommended for use only in the REPL.

As for how SystemExit is handled, it eventually causes the VM to call os._exit, but before that, it does some cleanup. It also runs atexit._run_exitfuncs() which runs any callbacks registered via the atexit module. Calling os._exit directly bypasses the atexit step.

so, raise SystemExit may be the preferable way to exit when the exception is caught.

I tried it with `raise SystemExit` instead of `os._exit(0)`. That made it much worse. With `raise SystemExit` then the program only works with 1 thread (the main thread only). At least with `os._exit(0)` I can get 25 threads. — personal_cloud, Sep 12 '17 at 16:26
Can you add an edit to your question showing what you did when you replaced `os._exit(0)` with `raise SystemExit`? It seems inconceivable that a 1-to-1 exchange of them would change the number of allowable threads, but if you say it does - then it'd be interesting to test your code in my environment to verify this strange phenomenon. — boardrider, Sep 13 '17 at 17:40
Why is it strange for `raise SystemExit` to behave any differently from `raise KeyboardInterrupt`? Given that the latter is broken (otherwise this whole question wouldn't exist) I am not surprised about the former being broken too. — personal_cloud, Sep 15 '17 at 17:14

python 2.7: how to catch keyboard interrupt in program with more than 25 threads

2 Answers2

Linked