1

I can't find any reference to this question and it seems so trivial.

Is it even possible to share objects across Pythons workers from multiprocessing Pool ?

Here's a little example:

from multiprocessing import Pool

def work(a):
    return do_work(obj_b)

def main(obj_a, obj_b):
    my_iterable = get_iter(obj_a)
    p = Pool(processes=6)
    res = p.map(work, my_iterable)

Lets say get_iter(obj_a) returns an iterable object. How does "work" know of obj_b?

Eran Moshe
  • 3,062
  • 2
  • 22
  • 41

2 Answers2

2

After reading a lot of material I've come to realize a few things:

  1. Python isn't commonly used for multiprocessing.
  2. The so called "Shared Object" is probably (and I'm not sure of it, I'll be happy to stand corrected) is being fully copied by every process.
  3. It works :>

Heres the code:

from multiprocessing import Pool, cpu_count

def work(a):
    print("I'm aware of obj_b: {}".format(obj_b))

def initPoolResources(_obj_b):
    # Define all your shared read obj here
    global obj_b 
    # Initialize them
    obj_b = _obj_b

def main(obj_a):
    # Assume obj_a is an iterable object
    # We want to create a "shared read only" object between the pool of processes. 
    p = Pool(processes=cpu_count()-1, initializer=initPoolResources, initargs(obj_b))
    result = p.map(work, obj_a)
    p.close()
    p.join()

work(a) has never seen obj_b, but he's fully aware of it.

Eran Moshe
  • 3,062
  • 2
  • 22
  • 41
1

Yes it is possible, from the doc. You can create a shared object and if set globally you can do it. See this SO answer.

tupui
  • 5,738
  • 3
  • 31
  • 52
  • This is shared across Process. I want to share across a Pool() of workers. – Eran Moshe Jan 21 '18 at 17:02
  • your 2nd attached link has helped me a lot. I will write the final solution and if you'd like you can copy pasta it and I'll accept it as the answer. Otherwise, I'll accept it as the answer because it will make things much more clean and straight forward. – Eran Moshe Jan 25 '18 at 08:54
  • You’re welcome. Don’t worry keep your answer ;) do +1 if it helped – tupui Jan 25 '18 at 10:07