Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
112
votes
4 answers

How does ThreadPoolExecutor().map differ from ThreadPoolExecutor().submit?

I was just very confused by some code that I wrote. I was surprised to discover that: with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor: results = list(executor.map(f, iterable)) and with…
Patrick Collins
  • 10,306
  • 5
  • 30
  • 69
94
votes
5 answers

Use tqdm with concurrent.futures?

I have a multithreaded function that I would like a status bar for using tqdm. Is there an easy way to show a status bar with ThreadPoolExecutor? It is the parallelization part that is confusing me. import concurrent.futures def f(x): return…
max
  • 4,141
  • 5
  • 26
  • 55
64
votes
4 answers

How do you kill Futures once they have started?

I am using the new concurrent.futures module (which also has a Python 2 backport) to do some simple multithreaded I/O. I am having trouble understanding how to cleanly kill tasks started using this module. Check out the following Python 2/3 script,…
Nick Chammas
  • 11,843
  • 8
  • 56
  • 115
58
votes
2 answers

What is the difference between concurrent.futures and asyncio.futures?

To clarify the reason for this question: It is confusing to use two modules with the same name. What do they represent that makes them distinct? What task(s) can one solve that the other can't and vice-versa?
sargas
  • 5,820
  • 7
  • 50
  • 69
53
votes
1 answer

ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool

I was experimenting with the new shiny concurrent.futures module introduced in Python 3.2, and I've noticed that, almost with identical code, using the Pool from concurrent.futures is way slower than using multiprocessing.Pool. This is the version…
astrojuanlu
  • 6,744
  • 8
  • 45
  • 105
37
votes
2 answers

Finding the cause of a BrokenProcessPool in python's concurrent.futures

In a nutshell I get a BrokenProcessPool exception when parallelizing my code with concurrent.futures. No further error is displayed. I want to find the cause of the error and ask for ideas of how to do that. Full problem I am using…
Samufi
  • 2,465
  • 3
  • 19
  • 43
36
votes
1 answer

How does concurrent.futures.as_completed work?

I'm learning about python concurrency and I was introduced with the concept of futures. I read that as_completed() takes an iterable of futures and yields them as they are done. I want to know how it works internally. Is it yielding completed tasks…
blueplusgreen
  • 487
  • 1
  • 5
  • 9
36
votes
1 answer

Number of max_workers when using ThreadPoolExecutor from concurrent.futures?

What are the factors to consider when deciding what to set max_workers to in ThreadPoolExecutor from concurrent.futures? As long as you can expect Python 3.5+ to be available, is there any reason not to set max_workers to None which will then…
edjohnsonwilliams
  • 461
  • 1
  • 4
  • 4
34
votes
2 answers

python concurrent.futures.ProcessPoolExecutor: Performance of .submit() vs .map()

I am using concurrent.futures.ProcessPoolExecutor to find the occurrence of a number from a number range. The intent is to investigate the amount of speed-up performance gained from concurrency. To benchmark performance, I have a control - a serial…
Sun Bear
  • 7,594
  • 11
  • 56
  • 102
32
votes
2 answers

The workers in ThreadPoolExecutor is not really daemon

The thing I cannot figure out is that although ThreadPoolExecutor uses daemon workers, they will still run even if main thread exit. I can provide a minimal example in python3.6.4: import concurrent.futures import time def fn(): while True: …
Sraw
  • 18,892
  • 11
  • 54
  • 87
27
votes
4 answers

Exception handling in concurrent.futures.Executor.map

From https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.map If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator. The following snippet only outputs…
user1008636
  • 2,989
  • 11
  • 31
  • 45
26
votes
1 answer

Nesting concurrent.futures.ThreadPoolExecutor

I have a program where I am currently using a concurrent.futures.ThreadPoolExecutor to run multiple tasks concurrently. These tasks are typically I/O bound, involving access to local databases and remote REST APIs. However, these tasks could…
Andrew McLean
  • 383
  • 4
  • 11
26
votes
4 answers

Python: concurrent.futures How to make it cancelable?

Python concurrent.futures and ProcessPoolExecutor provide a neat interface to schedule and monitor tasks. Futures even provide a .cancel() method: cancel(): Attempt to cancel the call. If the call is currently being executed and cannot be…
Ketzu
  • 660
  • 1
  • 7
  • 12
26
votes
3 answers

Getting original line number for exception in concurrent.futures

Example of using concurrent.futures (backport for 2.7): import concurrent.futures # line 01 def f(x): # line 02 return x * x # line 03 data = [1, 2, 3, None, 5] # line 04 with concurrent.futures.ThreadPoolExecutor(len(data)) as executor: #…
djeendo
  • 317
  • 1
  • 3
  • 9
25
votes
2 answers

`DummyExecutor` for Python's `futures`

Python's futures package allows us to enjoy ThreadPoolExecutor and ProcessPoolExecutor for doing tasks in parallel. However, for debugging it is sometimes useful to temporarily replace the true parallelism with a dummy one, which carries out the…
Ram Rachum
  • 84,019
  • 84
  • 236
  • 374
1
2 3
…
61 62