Killing Thread and releasing memory in Python

Question

I'm killing thread like shown here: Is there any way to kill a Thread in Python?

But I noticed, that memory is not released (gc.get_objects() keeps growing and growing). In fact these objects are lists, dicts etc., not files.

Id there any way to manually release resources? Code:

import ctypes

def terminate_thread(thread):
    """Terminates a python thread from another thread.

    :param thread: a threading.Thread instance
    """
    if not thread.isAlive():
        return

    exc = ctypes.py_object(SystemExit)
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(
        ctypes.c_long(thread.ident), exc)
    if res == 0:
        raise ValueError("nonexistent thread id")
    elif res > 1:
        # """if it returns a number greater than one, you're in trouble,
        # and you should call it again with exc=NULL to revert the effect"""
        ctypes.pythonapi.PyThreadState_SetAsyncExc(thread.ident, None)
        raise SystemError("PyThreadState_SetAsyncExc failed")

class MyThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.result = None
        self.error = None

    def run(self):
        try:
            self.result = myfun(*args, **kw) #run external resource and the interrupt it
        except Exception as e:
            self.error = e

c = MyThread()
c.start()
c.join(60) # wait a minute
counter = 0
if c.isAlive():
    while c.isAlive():
        time.sleep(0.1)
        try:
            terminate_thread(c) # how to release resources?
        except:
            break
        counter += 1
        if counter > 10: break
    raise TimeoutException

Example of output of:

print('Controlled objects: %s' % len(gc.get_objects()))
print ('Unreachable: %s' % gc.collect())

Controlled objects: 85084 #request 1, no timeout
Unreachable: 5640

Controlled objects: 171994 # request 2, timeout
Unreachable: 7221

please create an example that shows the problem with the minimum amount of code possible. the answer to your question as it stands is simply __no__ ... there is no way for the parent to kill the thread — Joran Beasley, Feb 18 '15 at 23:54
what you mean parent? I use example from link above, exactly as it is — Paul R, Feb 18 '15 at 23:59
you can only exit the thread from inside the thread .... I can only assume you are not correctly exiting from your thread .... — Joran Beasley, Feb 19 '15 at 00:11
the way shown(linked by you in original question) shows several "correct" ways .... so its clearly an implementation issue ... so we circle back around to "please create a minimal example that demonstrates this issue" — Joran Beasley, Feb 19 '15 at 00:18
you are getting closer ... there is still no myfun .... that is being called ... as it stands this should exit the thread no problem due to an error that myfun does not exist — Joran Beasley, Feb 19 '15 at 00:39
myfun is external library. can't copy it here. assume that it works well, but sometimes works longer than desired, so needs to be interrupted. — Paul R, Feb 19 '15 at 00:42
so you need to provide an example that does that ... as it stands i cant reproduce your problem ... (while your at it you should probably add debug print statements to prove that you have a memory leak ) — Joran Beasley, Feb 19 '15 at 00:43
if `myfun` is external, then note the comment in the example you are copying: `As noted in the documentation, this is not a magic bullet because if the thread is busy outside the Python interpreter, it will not catch the interruption.` — Mark Tolonen, Feb 19 '15 at 00:44
See update on output. I need to interrupt external function. Is there reliable way. Can't use signals, subprocesses because of multithreaded mod_wsgi. — Paul R, Feb 19 '15 at 00:50
why does that explain why you cannot use subprocess or signals? (Im sure its obvious if you use mod_wsgi alot ...) — Joran Beasley, Feb 19 '15 at 00:53
Don't know. Just doesn't work (doesn't raise TimeoutException). Will inverstigate further. — Paul R, Feb 19 '15 at 00:54
if you are using threads and it is doing something external to your loop for 12 minutes theres not alot you can do ... maybe edit `myfun` to check for conditions to exit at some point ... you wont check the exit condition of your loop until its next iteration ... you however dont even have a loop ... so there is no exit condition to check ... w — Joran Beasley, Feb 19 '15 at 16:50

score 4 · Accepted Answer · edited Jun 20 '20 at 09:12

ok after all this garbage I think what you want is the multiprocessing module as I believe you can actually send a sigkill on that

class MyThread:
    def __init__(self):
        self.result = None
        self.error = None
    def start(self):
        self.proc = multiprocessing.Process(target=self.run)
        self.proc.start()
    def stop(self):
       self.proc.send_signal(multiprocessing.SIG_KILL)
    def run(self):
        try:
            self.result = myfun(*args, **kw) #run external resource and the interrupt it
        except Exception as e:
            self.error = e

then you would call c.stop() in order to halt the thread with the sig_kill (of coarse the other thing should respond appropriately to this)

you can probably even just use the builtin subprocess.Process.kill() (see the docs https://docs.python.org/2/library/subprocess.html#subprocess.Popen.send_signal)

WRT YOUR QUESTION##

(Id there any way to manually release resources? )

 t = Thread(target=some_long_running_external_process)
 t.start()

there is no way to exit your thread (t) from outside of some_long_running_external_process

And you beat me to this :) There is also a `.terminate()` method in multiprocessing. — Mark Tolonen, Feb 19 '15 at 00:50

Killing Thread and releasing memory in Python

1 Answers1

WRT YOUR QUESTION##