0

I am working on a code where threads are generated in a recursive function. So its tedious to follow all threads. Thread.join() is not feasible. You can do it but with a lot of effort. What I want is to wait till all threads(child process) are completed, before printing result generated after running threads. I guess you can get the gist of the problem. All I want is to execute a statement just before exiting the main program.

martineau
  • 119,623
  • 25
  • 170
  • 301
KRISHNA I
  • 57
  • 2
  • 11
  • Can't you resort to appending threads to a threads list and then `[thread.join() for thread in threads]`? – Rodrigo Oliveira Jan 22 '19 at 16:53
  • Does [this](https://stackoverflow.com/questions/11968689/python-multithreading-wait-till-all-threads-finished) answer helps? – yorodm Jan 22 '19 at 17:00
  • 1
    You want to wait for threads to die. That's what `join()` _does_. How is it not "feasible?" Why can't you make a list? Every time you create a thread, add the `Thread` object to the list. When it's time to wait for all of the threads to die, iterate over the list and `join()` each one. – Solomon Slow Jan 22 '19 at 17:03
  • Or create a list of the threads and keep iterating over it until all are `not lst[i].is_alive()`. This is a bad practice called "busy waiting" however... – martineau Jan 22 '19 at 17:18
  • Re, "...threads are generated in a recursive function..." That sounds *very* suspicious. What are those threads doing? Why aren't you using a _[thread pool](https://en.wikipedia.org/wiki/Thread_pool)_? (as in, djoffe's answer, below) – Solomon Slow Jan 22 '19 at 18:51
  • martineau I mentioned that I can't track all threads. Its possible to do as you say if I can track threads. Tracking threads is difficult. It's not like I am using specific number of threads or starting them at a time. So using thread.join is not feasible – KRISHNA I Jan 23 '19 at 13:33
  • Solomon Slow thanks. I am new to python so not much knowledgeable. I will try using thread pool – KRISHNA I Jan 25 '19 at 17:19

1 Answers1

3

Did you try using the concurrent.futures package?

You can instantiate a ThreadPoolExecutor and start your threads by submitting to it. Then call the executor's shutdown(wait=True) function to wait for all threads to complete.

Alternatively, use a with ThreadPoolExecutor as e: statement. When you exit the with block, all your threads are completed.

djoffe
  • 102
  • 8
  • 1
    Note: This doesn't actually create a thread per task (it's a pool of thread workers). That said, you usually don't *want* a thread per task for a ton of small tasks, so this is likely better than the OP's proposed solution. – ShadowRanger Jan 22 '19 at 17:38
  • Shadow ranger can you elaborate what you mentioned? Are you talking about djoffe answer? – KRISHNA I Jan 23 '19 at 13:35
  • 3
    @KRISHNAI: Executors create a pool of workers (threads for `ThreadPoolExecutor`) immediately, all of which listen to a queue for work items. `submit` doesn't launch new threads, it just packages up the task and shoves it on the queue to be pulled by the first available worker. In practice, this is usually better than a thread per worker, since the cost of launching a thread is non-trivial, and for 1000+ work items, you might not be *able* to run that many threads simultaneously. – ShadowRanger Jan 24 '19 at 02:53
  • Oh! Thanks ShadowRanger – KRISHNA I Jan 25 '19 at 05:49