2

I have a list of class instances. Each of this class objects have a method function that I want to run and I want to parallelize across the class instances since there's a lot of them. For example, I have this class object:

class Person(object):
    def __init__(self, name):
        self.name = name
    def register(self):
        self.registered = True

Now I defined a list of class instances of Person()

persons = [Person(i) for i in ["Bernard", "Enrik", "Joseph"]]

I want to parallelize the running of Person.register() method function across the values in persons. Using multiprocessing.Pool, here's what I did. Since you can't pickle method function, I wrap it in another function.

import multiprocessing as mp
def reg(person):
    person.register()
pool = mp.Pool(processes=mp.cpu_count()//2)
res = list(pool.map(reg, persons))

The code runs without error by the way, so now I am expecting that every Person instance in the list will have an attribute registered set to True but when I check each class instance, it doesn't contain attribute registered. For example, checking if registered is an attribute of persons[0],

hasattr(persons[0], "registered")
>>> False

Why is this so? How can I solve this?

bninopaul
  • 2,659
  • 4
  • 15
  • 22
  • You have got the map. That is a good start. Now collect it in a container and be happy. `list(pool.map(reg, persons))` would be my first try as `list(map(reg, persons))` will set the attribute for all your `Person`s ;) – Uvar Oct 26 '17 at 12:56
  • Do you mean I should put a return statement in `Person.register()` and given the `list(pool.map(reg, persons))`, I should run another function that would explicitly assign the attribute `registered` to each class object? – bninopaul Oct 26 '17 at 13:20

1 Answers1

2

I found a similar problem here: Python multiprocessing with pathos. The reason why the Person class instance doesn't have attribute registered is that in the multiprocessing, the Pool creates processes that are just copies of the class instances in the list, thus, to solve this, in wrapper function reg() return the class instances, that is,

def reg(person):
    person.register()
    return person
pool = mp.Pool(processes=mp.cpu_count()//2)
persons = list(pool.map(reg, persons))
bninopaul
  • 2,659
  • 4
  • 15
  • 22
  • I am doing this with my own class, method and wrapper function, but i get something that would be for this code as follows: `_pickle.PicklingError: Can't pickle reg: attribute lookup reg on __main__ failed`, anyone know why? – someoneb100 Dec 13 '17 at 14:20
  • @someoneb100, did you make sure that wrapper reg() function is outside the class? – bninopaul Dec 13 '17 at 15:23
  • yes https://stackoverflow.com/questions/47795810/applying-a-method-in-a-list-of-class-objects-using-multiprocessing here is my problem – someoneb100 Dec 13 '17 at 15:47