For questions related to parallelism achieved through pools of worker processes.
Questions tagged [process-pool]
110 questions
101
votes
2 answers
Python multiprocessing.Pool: AttributeError
I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores.
I wrote the following code, which works if I use normal map(), but with pool.map() returns an error.
import…
Amit
- 5,924
- 7
- 46
- 94
47
votes
4 answers
Starmap combined with tqdm?
I am doing some parallel processing, as follows:
with mp.Pool(8) as tmpPool:
results = tmpPool.starmap(my_function, inputs)
where inputs look like:
[(1,0.2312),(5,0.52) ...]
i.e., tuples of an int and a float.
The code runs nicely, yet…
sdgaw erzswer
- 2,182
- 2
- 26
- 45
32
votes
4 answers
multiprocessing returns "too many open files" but using `with...as` fixes it. Why?
I was using this answer in order to run parallel commands with multiprocessing in Python on a Linux box.
My code did something like:
import multiprocessing
import logging
def cycle(offset):
# Do stuff
def run():
for nprocess in…
nephewtom
- 2,941
- 3
- 35
- 49
8
votes
1 answer
Submit code for execution to all processes in a concurrent.futures.ProcessPool
Context:
A Python application server that uses a concurrent.futures.process.ProcessPool to execute code
We sometimes want to hot reload imported code without restarting the entire server process
(yes I know importlib.reload has caveats)
To get…
codeape
- 97,830
- 24
- 159
- 188
8
votes
3 answers
os.sched_getaffinity(0) vs os.cpu_count()
So, I know the difference between the two methods in the title, but not the practical implications.
From what I understand: If you use more NUM_WORKERS than are cores actually available, you face big performance drops because your OS constantly…
rocksNwaves
- 5,331
- 4
- 38
- 77
7
votes
0 answers
Use tqdm with multiprocessing for multiple progress bars
I want to monitor progress across multiple workers which are different processes. For each subprocess I have its own progress bar but it doest work properly with ProcessPoolExecutor executor.
def main():
with…
Most Wanted
- 6,254
- 5
- 53
- 70
6
votes
1 answer
How to terminate long-running computation (CPU bound task) in Python using asyncio and concurrent.futures.ProcessPoolExecutor?
Similar Question (but answer does not work for me): How to cancel long-running subprocesses running using concurrent.futures.ProcessPoolExecutor?
Unlike the question linked above and the solution provided, in my case the computation itself is…
Darkfish
- 397
- 1
- 9
- 17
5
votes
1 answer
Why do ProcessPoolExecutor and Pool crash with a super() call?
1. Why does the following Python code using the concurrent.futures module hang forever?
import concurrent.futures
class A:
def f(self):
print("called")
class B(A):
def f(self):
executor =…
Géry Ogam
- 6,336
- 4
- 38
- 67
4
votes
0 answers
Capture / redirect all output of ProcessPoolExecutor
I am trying to capture all output from a ProcessPoolExecutor.
Imagine you have a file func.py:
print("imported") # I do not want this print in subprocesses
def f(x):
return x
then you run that function with a ProcessPoolExecutor like
from…
johnbaltis
- 1,413
- 4
- 14
- 26
4
votes
2 answers
Are results of multiprocessing.Pool.map_async() returned in the same order of the input?
Reading multiprocessing.Pool doc I understood that map_async and apply_async are two versions of map and appy that are supposed to be faster, but that do not guarantee that the inputs are processed in the same order as they are provided.
However, I…
Luca
- 1,610
- 1
- 19
- 30
4
votes
1 answer
Incorrect number of running calls with ProcessPoolExecutor in Python
In Python's concurrent.futures standard module, why does the number of running calls in a ProcessPoolExecutor is max_workers + 1 instead of max_workers like in a ThreadPoolExecutor? This happens only when the number of submitted calls is strictly…
Géry Ogam
- 6,336
- 4
- 38
- 67
4
votes
5 answers
How to count word frequencies in a huge file concurrently?
I need to count word frequency of a 3GB gzipped plain text file of English sentences, which is about 30 GB when unzipped.
I have a single thread script with collections.Counter and gzip.open, it takes hours to finish.
Since reading a file line by…
Galaxy
- 1,862
- 1
- 17
- 25
4
votes
2 answers
Is there any way to use multiprocessing.pool within a nested function or module?
thanks for taking a look at this. I confess I have been dabbling with parallel processing in python for all of 1 week now so I apologize if there is an obvious solution I missed. I have a piece of code that I would like to run several different…
matrimcauthon
- 63
- 1
- 5
4
votes
4 answers
Python doctest hangs using ProcessPoolExecutor
This code runs fine under regular CPython 3.5:
import concurrent.futures
def job(text):
print(text)
with concurrent.futures.ProcessPoolExecutor(1) as pool:
pool.submit(job, "hello")
But if you run it as python -m doctest myfile.py, it…
John Zwinck
- 239,568
- 38
- 324
- 436
3
votes
1 answer
Multiprocess pool initialization with sequential initializer argument
I have some code like the following:
import multiprocessing as mp
connection: module.Connection
def client_id():
for i in range(mp.cpu_count*2):
yield i
def initproc(host: str, port: int, client_id: int):
global connection
…
Karlson
- 2,958
- 1
- 21
- 48