2

Following code not executing as expected.

import multiprocessing

lock = multiprocessing.Lock()
def dummy():
    def log_results_l1(results):
        lock.acquire()
        print("Writing results", results)
        lock.release()

    def mp_execute_instance_l1(cmd):
        print(cmd)
        return cmd

    cmds = [x for x in range(10)]

    pool = multiprocessing.Pool(processes=8)

    for c in cmds:
        pool.apply_async(mp_execute_instance_l1, args=(c, ), callback=log_results_l1)

    pool.close()
    pool.join()
    print("done")


dummy()

But it does work if the functions are not nested. What is going on.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
kanna
  • 1,412
  • 1
  • 15
  • 33
  • what's the type of `lock` ? – Jean-François Fabre Sep 17 '17 at 17:19
  • For the future, while your problem was obvious enough from context, you can't just say "it doesn't work". You need to provide a [MCVE], and in this case, that would include the error and the exception traceback that occurred on failure. – ShadowRanger Sep 17 '17 at 17:24

1 Answers1

5

multiprocessing.Pool methods like the apply* and *map* methods have to pickle both the function and the arguments. Functions are pickled by their qualified name; essentially, on unpickling, the other process needs to be able to import the module they were defined in and do a getattr call to find the function in question. Nested functions aren't available by name outside the function they were defined in, so pickling fails. When you move the function to global scope, you fix this, which is why it works when you do that.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271