How to handle out-of-memory error while using threads in python

Question

I have memory leak, but I can't find a way to solve it. I think the reason is for that because I use threads and don't stop/kill it in a right way.

I have following method:

import threading
def worker():
     if nextJobActive() and number_of_active_threads<5:
         t = threading.Thread(target=startThread, args=(my_list, my_item))
         t.start() 

def startThread(): 
    #do something here, which takes ~15 Min.

I run the worker() method in while(true) loop. I always have to start new threads in my case. But I never stop a thread. I also don't know how to do this. Is there anyway to safely stop a thread in my case?

score 1 · Answer 1 · answered Jun 08 '21 at 22:33

1

As you know already, you are creating an endless amount of threads without properly stopping the previous one. To wait for a thread to terminate there is a .join() method. Here is the documentation for the Thread module: docs.

import threading
def worker():
     if nextJobActive() and number_of_active_threads<5:
         t = threading.Thread(target=startThread, args=(my_list, my_item))
         t.start()
         t.join() 

def startThread(): 
    #do something here, which takes ~15 Min.

answered Jun 08 '21 at 22:33

VRComp

131
1
1
12

This means, new threads will always be started and the old ones will be terminated when their job is done? – Cenk Ten Jun 09 '21 at 06:29
I just tested it; without join it runs always 5 threads parallel, but with join it runs at first 5 but then only 1 thread. Any thoughts why? – Cenk Ten Jun 09 '21 at 06:58
The one thread is the main thread that is running. The first five threads execute then they each waits to be terminated then the main program continues running as the main thread and only thread. If you plan to achieve multithreading then you can use lock in the threading module or concurrent.futures module (https://docs.python.org/3/library/concurrent.futures.html) – VRComp Jun 09 '21 at 11:04
Thank you for the answer. In my case I have to use threading, replacing it with concurrent will cause a big overhead. What if I kill the thread when its job is done manually? Or thoughts if it will work for me? – Cenk Ten Jun 09 '21 at 12:08
Killing a thread in Python is dangerous. The easiest way is to execute your thread as daemon and it will keep running as the main thread finishes and shutdown. This might help: https://stackoverflow.com/questions/11431637/how-can-i-kill-a-thread-in-python – VRComp Jun 09 '21 at 13:00
My main program (main thread?) stops never, that's why its generating always new threads. You mean if I set ```thread.daemon = True``` it will help me? but in this case when does it stop? after all jobs are done? actually my main method ```startThread()``` call the command return, I wonder why my thread still is running after calling return... – Cenk Ten Jun 09 '21 at 14:16
And I wonder why sys.exit() in thread that are finished their work. If I handle the exception, this also should work for me? – Cenk Ten Jun 09 '21 at 14:48
1

Your main program is your main thread. Since your main program never stops calling return won't help. If you can handle all exceptions and know which threads finish their work then killing a thread should work. – VRComp Jun 09 '21 at 15:11
Thank you very much. – Cenk Ten Jun 09 '21 at 22:00

How to handle out-of-memory error while using threads in python

1 Answers1