Python2 killing a thread by PyThreadState_SetAsyncExc

Question

I have a blocking function that I need to kill after a certain amount of time. But I have to use python2.7 for 'reasons' (sorry). I am using this to achieve what I need/

def fn(v):
    try:
        for i in range(v):
            print(i)
            time.sleep(1)
    except:
        print('exc!')
        raise
    print('done...')

t = threading.Thread(target=fn, args=(5,))
t.start()
time.sleep(3)

if t.is_alive():
    print("killing", t.ident)
    ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(t.ident), ctypes.py_object(SystemExit))
time.sleep(3)

I let the function block for 5 seconds but I want the thread to be killed after 3 seconds. And it works like a charm in python3. But in python2, the function does not get killed and runs its course.

If I replace SystemExit with RuntimeError (to get a more descriptive output), this is what I get in python3, which I expect

0
1
2
killing 139879494002432
exc!  # <-- NICE!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "srv.py", line 68, in fn
    time.sleep(1)
RuntimeError

But this is python2. It raises the exception but only after finishing the thread for some reason

0
1
2
killing 139903361537792
3
4
done  # <-- I DON'T NEED THIS!
Unhandled exception in thread started by <bound method Thread.__bootstrap of <Thread(Thread-1, stopped 139903361537792)>>
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 849, in __bootstrap_inner
    self.__stop()
  File "/usr/lib/python2.7/threading.py", line 864, in __stop
    self.__block.notify_all()
  File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
    self.notify(len(self.__waiters))
  File "/usr/lib/python2.7/threading.py", line 383, in notify
    if not self._is_owned():
RuntimeError

Any help is highly appreciated.

CristiFati · Answer 1 · 2022-08-19T18:53:44.660

According to [GitHub.Gist]: liuw/ctype_async_raise.py - Nasty hack to raise exception for other threads (@wyl8899's comment):

I believe this is because the exception is "async" since the API is _SetAsyncExc.

Since the thread probably spends more than 99.99%, in time.sleep, that time should be fragmented in order to give the thread a chance to process the exception.
I didn't check why it behaves differently in the 2 Python versions (as there are major changes between them), or which behavior is the "correct" one, but this should work on each of them.

A common mistake when working with CTypes is explained here: [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer).

code00.py:

#!/usr/bin/env python

import ctypes as ct
import sys
import threading
import time

try:  # @TODO - cfati: Don't do this in production! (Python 3 / 2)
    import _thread as thread
except:
    import thread


get_ident = thread.get_ident


def fragmented_sleep(seconds, intervals=1):
    intervals = max(intervals, 1)
    ts = float(seconds) / intervals
    time_end = time.time() + seconds
    remaining = time_end - time.time()
    while remaining > 0:
        time.sleep(min(ts, remaining))
        remaining = time_end - time.time()


def func(count):
    print("  Thread 0x{:016X} start".format(get_ident()))
    try:
        for i in range(count):
            print("  Iteration {:d}".format(i))
            fragmented_sleep(1, intervals=10)
    except:
        print("  Thread 0x{:016X} exception!!!".format(get_ident()))
        raise
    print("  Thread 0x{:016X} end gracefully".format(get_ident()))


def main(*argv):
    ptssae = ct.pythonapi.PyThreadState_SetAsyncExc
    ptssae.argtypes = (ct.c_ulong, ct.py_object)
    ptssae.restype = ct.c_int

    t = threading.Thread(target=func, args=(5,))
    t.start()
    time.sleep(3)
    if t.is_alive():
        print("Attempting to kill thread 0x{:016X}".format(t.ident))
        res = ptssae(t.ident, RuntimeError)
        print("Attempt result: {:d}".format(res))
    print("Exit main")


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

Nix:

(qaic-env) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q073413798]> python ./code00.py
Python 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] 064bit on linux

  Thread 0x00007FB2B2453700 start
  Iteration 0
  Iteration 1
  Iteration 2
  Iteration 3
Attempting to kill thread 0x00007FB2B2453700
Attempt result: 1
Exit main

Done.
  Thread 0x00007FB2B2453700 exception!!!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "./code00.py", line 32, in func
    fragmented_sleep(1, intervals=10)
  File "./code00.py", line 23, in fragmented_sleep
    time.sleep(min(ts, remaining))
RuntimeError
(qaic-env) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q073413798]> python2 ./code00.py
Python 2.7.18 (default, Jul  1 2022, 12:27:04) [GCC 9.4.0] 064bit on linux2

  Thread 0x00007F6F02E68700 start
  Iteration 0
  Iteration 1
  Iteration 2
  Iteration 3
Attempting to kill thread 0x00007F6F02E68700
Attempt result: 1
Exit main

Done.
  Thread 0x00007F6F02E68700 exception!!!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./code00.py", line 32, in func
    fragmented_sleep(1, intervals=10)
  File "./code00.py", line 24, in fragmented_sleep
    remaining = time_end - time.time()
RuntimeError

Win:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q073413798]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" ./code00.py
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32

  Thread 0x0000000000009DD4 start
  Iteration 0
  Iteration 1
  Iteration 2
Attempting to kill thread 0x0000000000009DD4
Attempt result: 1
Exit main

Done.
  Thread 0x0000000000009DD4 exception!!!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\Install\pc064\Python\Python\03.09\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "c:\Install\pc064\Python\Python\03.09\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "e:\Work\Dev\StackOverflow\q073413798\code00.py", line 32, in func
    fragmented_sleep(1, intervals=10)
  File "e:\Work\Dev\StackOverflow\q073413798\code00.py", line 23, in fragmented_sleep
    time.sleep(min(ts, remaining))
RuntimeError

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q073413798]> "e:\Work\Dev\VEnvs\py_pc064_02.07.18_test0\Scripts\python.exe" ./code00.py
Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 20 2020, 13:25:05) [MSC v.1500 64 bit (AMD64)] 064bit on win32

  Thread 0x00000000000075A0 start
  Iteration 0
  Iteration 1
  Iteration 2
Attempting to kill thread 0x00000000000075A0
Attempt result: 1  Iteration 3

Exit main

Done.
  Thread 0x00000000000075A0 exception!!!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\Install\pc064\Python\Python\02.07.18\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "c:\Install\pc064\Python\Python\02.07.18\lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./code00.py", line 31, in func
    print("  Iteration {:d}".format(i))
RuntimeError

As I understand, in `python3`, when we raise an exception in subthread, this excepion will be pending until subthread will check it. That's why blocking operation will prevent raise an exception in blocked thread. However `python2` has another approach I think, but I don't understand what it is doing. For example when `time.sleep(1)` expires, then `python2` should check a pending exception and action for it but it is not doing this or there is another thing that I don't know — Veysel Olgun, Aug 19 '22 at 23:32
Is there a way to make it work for the blocking input() call too (in python 3)? — MMM, Dec 23 '22 at 06:48

Python2 killing a thread by PyThreadState_SetAsyncExc

1 Answers1