Questions tagged [multiprocessing-manager]
88 questions
16
votes
2 answers
Python manager.dict() is very slow compared to regular dict
I have a dict to store objects:
jobs = {}
job = Job()
jobs[job.name] = job
now I want to convert it to use manager dict because I want to use multiprocessing and need to share this dict amonst processes
mgr = multiprocessing.Manager()
jobs =…
ealeon
- 12,074
- 24
- 92
- 173
15
votes
1 answer
Sharing object (class instance) using multiprocessing Managers
I need to share an object and its methods between several processes in python. I am trying to use managers (in module multiprocessing) but it crashes. Here is a silly example of producer-consumer where the shared object between the two processes is…
SagittariusA
- 5,289
- 15
- 73
- 127
14
votes
2 answers
Unable to update nested dictionary value in multiprocessing's manager.dict()
I am trying to update a key in a nested dictionary of multiprocessing module's manager.dict() but not able to do so. It doesn't update the value and doesn't throw any error too.
Code:
import time
import random
from multiprocessing import Pool,…
MohitC
- 4,541
- 2
- 34
- 55
6
votes
2 answers
Python 3.6+: Nested multiprocessing managers cause FileNotFoundError
So I'm trying to use multiprocessing Manager on a dict of dicts, this was my initial try:
from multiprocessing import Process, Manager
def task(stat):
test['z'] += 1
test['y']['Y0'] += 5
if __name__ == '__main__':
test =…
mewais
- 1,265
- 3
- 25
- 42
6
votes
5 answers
Rate Limit Downloads Amongst Multiple Processes
I want to download and process a lot of files from website. The terms of service for the site restrict the number of files you're permitted to download per second.
The time that it takes to process the files is actually the bottle neck, so I'd like…
Batman
- 8,571
- 7
- 41
- 80
5
votes
0 answers
Python Multiprocessing fails in first attempt of command execution in connected devices, From second attempt onwards passed in all subprocesses
I'm trying to run bunch of linux commands on multiple devices parallely using python multiprocessing module.
script is able to connect to multiple devices serially and starts executing the linux commands parallely.
the script fails in the first…
Venkat
- 51
- 3
5
votes
1 answer
How python manager.dict() locking works:
A managers.dict() allow to share a dictionary across process and perform thread-safe operation.
In my case each a coordinator process create the shared dict with m elements and n worker processes read and write to/from a single dict key.
Do…
Lorenzo Belli
- 1,767
- 4
- 25
- 46
4
votes
1 answer
Synchronization of writing to shared memory (list) in Python multiprocessing
I have the following code:
import multiprocessing
manager = multiprocessing.Manager()
Function that appends list if it's length is less than 4 or creates a new one with initial value 'y'.
def f(my_array):
if len(my_array) < 4:
…
4
votes
1 answer
python multiprocessing module, shared multidimensional array
I have a code which given two parameters, (k, m) will return a 4d numpy array, my requirement is that I need to calculate this array for possible values of (k,m) with k,m < N and add them up. This is slow in serial so I am trying to learn the…
user231042
- 63
- 2
2
votes
1 answer
Update shared dictionary using mpire package
I am working to update a shared dictionary synchronously using mpire package in Python in a multi-core machine (i.e., parallel processing to update a dict). The environment I am using is a Linux machine with 8 vCPU and 16 GB memory in Amazon…
Sauvik De
- 121
- 3
2
votes
1 answer
How to properly share Manager dict between processes
What I would like to do is to share a dictionary between subclasses of Process and when one process updates the dictionary the other is notified to use it. This is illustrated in the code below where MyProducer starts filling the dictionary and in…
dpservis
- 31
- 3
2
votes
0 answers
How to construct proxy objects from multiprocessing.managers.SyncManager?
I have long running file I/O tasks which I'd like to be able to move into a daemon/server process. A CLI tool would be used to queue new jobs to run, query the status of running jobs, and wait for individual jobs. Python's multiprocessing.managers…
toojays
- 787
- 6
- 10
2
votes
1 answer
Sharing mutable global variable in Python multiprocessing.Pool
I'm trying to update a shared object (a dict) using the following code. But it does not work. It gives me the input dict as an output.
Edit: Exxentially, What I'm trying to achieve here is to append items in the data (a list) to the dict's list.…
CKM
- 1,911
- 2
- 23
- 30
2
votes
0 answers
why multiprocessing.Queue() doesn't work but Manger.Queue() works in case of passing queue object to multiprocessing.Pool
As mentioned in the SO questions:
Can I pass queue object in multiprocessing pool starmap method
Sharing a result queue among several processes
when using Pool method of multiprocessing, multiprocessing.Queue() doesn't work and gives error when…
Tushar Seth
- 563
- 7
- 15
2
votes
0 answers
How to parallelize a class method or share class object between processes in python?
I have created a class containing all its instances and need to parallelize the process of instantiation, but cannot solve the problem of sharing the class as a class object. Is it possible in python 2.7 using multiprocessing?
OUTPUT_HEADINGS =…
Petr Krampl
- 816
- 9
- 11