1

Monkeypatching and multiprocessing. I have a class and I need to replace the object method by some other function (the idea of monkeypatching and I must do this because I will only know which function will be replaced at the runtime), then I need to use the multiprocessing for all the objects to call these function and do something (modify the value of attribute).

A simple example: I have a simple class called Test which has one attribute and one method called func

class Test:

  def __init__(self):
      self.attribute = 1

  def func(self):
      print("bye")

create two objects and put them in a list

a = Test()
b = Test()
c = [a, b]

define another function call func2

def func2(obj):
    obj.attribute = obj.attribute + 1
    print(obj.attribute)
    print("hi")

replace the fun in object a by func2

a.func = types.MethodType(func2, a)

run the multiprocessing

def func123(x):
   c[x].func()
   return c[x]


if __name__ == '__main__':
    pool = multiprocessing.Pool()
    d = pool.map(func123, range(2))
    pool.close()

Exception in thread Thread-3: AttributeError: 'Test' object has no attribute 'func2'

If I don't use multiprocessing and just simple call a.func, everything works fine. Also, If there is no return in the func123, the codes will work without error but it will not update the attribute value for object a.

叶圣晗
  • 11
  • 1
  • Could it be the case that on runtime, your `func2` function is getting accessed before it is there, I made a script of your code and it is working fine except for pickling error. – Pratik Oct 21 '19 at 09:34
  • See the answer here (https://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-multiprocessing-pool-map): `The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable` – ExplodingGayFish Oct 21 '19 at 09:42
  • @ExplodingGayFish Thanks for the reply, The error I got is in recv return _ForkingPickler.loads(buf.getbuffer()) AttributeError: 'Test' object has no attribute 'func2' instead of PicklingError: Can't pickle , which confused me. Also, I am not using multiprocessing inside a class and I am using multiprocessing to call a class method to do something and update that object. Without the monkeypatching, it works. The problem is the object "a" has changed (func is replaced by func2) – 叶圣晗 Oct 21 '19 at 10:17

1 Answers1

0

The bound attribute need in main point.

if __name__ == '__main__':
    a.func = types.MethodType(func2, a)
    pool = multiprocessing.Pool()
    d = pool.map(func123, range(2))
    pool.close()
DustyPosa
  • 463
  • 2
  • 8
  • The code is running without error but the attribute for object a is still not updated and it printd "bye" which means the monkey patching is not working properly. Also if I run this code on linux, I guess I don't need if __name__ == '__main__': Correct me if I am wrong. Many thanks for your help – 叶圣晗 Oct 21 '19 at 09:46