0

I have scripts running 24/7 that sometimes get stuck when a thread in concurrent.futures gets no response for a request. The hanging-threads 2.0.5 module prints out which thread hangs and why.

The print looks something like this:

Thread 139646566659840 "ThreadPoolExecutor-666849_1" hangs -                                                                                                                                                                                     
        File "/usr/lib/python3.9/threading.py", line 912, in _bootstrap                                                                                                                                                                          
                self._bootstrap_inner()                                                                                                                                                                                                          
        File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner                                                                                                                                                                    
                self.run()                                                                                                                                                                                                                       
        File "/usr/lib/python3.9/threading.py", line 892, in run                                                                                                                                                                                 
                self._target(*self._args, **self._kwargs)                                                                                                                                                                                        
        File "/usr/lib/python3.9/concurrent/futures/thread.py", line 77, in _worker                                                                                                                                                              
                work_item.run()                                                                                                                                                                                                                  
        File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run                                                                                                                                                                  
                result = self.fn(*self.args, **self.kwargs)

How can I, instead of just printing out the hanging threads and files, raise an exception when a thread is not responding in a given time? The script should just restart itself if hanging occurs, instead of waiting for a response.

I have tried with timeout, but concurrent futures can not be cancelled while running.

khelwood
  • 55,782
  • 14
  • 81
  • 108
PyNoob
  • 9
  • 3

1 Answers1

0

concurrent futures can not be cancelled while running

This is your problem. A hanging thread is still 'running'. Cancelling it from outside is not possible.

Thus you have two options:

  • switch to something which can be cancelled, like a ProcessPoolExecutor, or
  • rewrite the blocking code so it fails.

Since you say 'response to a request'---if this is a network request and you are early enough/frustrated enough in the dev cycle I thoroughly recommend switching to a concurrent multiprocessing framework like asyncio. This is exactly what they were developed for. In particular you may be interested in trios implementation of cancel scopes.

2e0byo
  • 5,305
  • 1
  • 6
  • 26
  • Thanks for your answer! I'd be happy to dig into these, but would be nice to know which option might be the best for me: I use concurrent.futures Threadpools to simultaneously send requests to a Broker. So, every thread is something simple like a function that requests the account balance or sends an order. I need those Threads either executed or failed in order to move on, because every further action requires the data acquired through the threads before. Is it even possible to make a thread fail if it is already waiting for response? – PyNoob Oct 26 '22 at 18:28
  • @pynoob I think the real question here is "how do I make this code nonblocking". That's a good separate question to ask. – 2e0byo Oct 26 '22 at 22:19