How to call a function that is inside another function using pool multiprocessing?

Question

I need to run the same function 10 times that for reasons of data linked to a login, it needs to be inside another function:

from multiprocessing import Pool

def main():
    def inside(a):
        print(a)
    Pool.map(inside, 'Ok' * 10)

if __name__ == '__main__':
    main()

from multiprocessing import Pool

def main():
    def inside(a):
        print(a)
    Pool.map(main.inside, 'Ok' * 10)

if __name__ == '__main__':
    main()

In both attempts the result is this:

AttributeError: 'function' object has no attribute 'map'

How can I do this by keeping the function inside the other function?
Is there a way to do this?

the error is because .map is not an attribute/method of Pool - Pool is a function (technically a method) — Derek Eden, Apr 26 '22 at 13:49
You have to invoke `.map()` on an *instance* of `Pool`, rather than the class itself. But I don't think this will work even after you fix that problem - the function being run has to have a globally-defined name, so that a reference to it can be transferred to a different process, and that's simply not true for a nested function. — jasonharper, Apr 26 '22 at 13:52
Hi friends, thanks for the comments. Can help me think of an option that would work for my need for a function within another function? I couldn't find an option. Because of the data created for ```API Client```, I can't put this function outside the other function. I even have a question that explains why I can't use it: https://stackoverflow.com/questions/71890447/error-when-trying-to-send-data-from-apiclient-to-a-function-using-multiprocessin — Digital Farmer, Apr 26 '22 at 13:55

Udesh · Accepted Answer · 2022-04-26T14:19:54.533

AttributeError: 'function' object has no attribute 'map'

We need to instantiate Pool from multiprocessing and call map method of that pool object.

You have to move inside method to some class because Pool uses pickel to serialize and deserialize methods and if its inside some method then it cannot be imported by pickel.

Pool needs to pickle (serialize) everything it sends to its worker-processes (IPC). Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. For that to work, the function needs to be defined at the top-level, nested functions won't be importable by the child and already trying to pickle them raises an exception (more).

Please visit this link of SO.

from multiprocessing import Pool

class Wrap:
    def inside(self, a):
        print(a)

def main():
    pool = Pool() 
    pool.map(Wrap().inside, 'Ok' * 10)

if __name__ == '__main__':
    main()

If you don't want to wrap inside method inside of a class move the inside method to global scope so it can be pickled

from multiprocessing import Pool

def inside(a):
    print(a)

def main():
    with Pool() as pool:
        pool.map(inside, 'Ok'*10)

if __name__ == '__main__':
    main()

How to call a function that is inside another function using pool multiprocessing?

1 Answers1