Pool issue when called from parent pool Python

Question

test1.py/myfunc1() does some work in parallel.

If I call myfunc1() from test2.py - it works fine (currently commented out).

If I create another pool in test2.py and call myfunc1() from those I get an unreported error in test1.py on the "pool = mp.Pool(5)" line .

result = {type} <class 'AssertionError'> args = {getset_descriptor} <attribute 'args' of 'BaseException' objects>

How do I fix this issue?

test1.py

import time
import multiprocessing as mp

def worker(a):
    print("Worker: "+str(a))
    time.sleep(5)
    return a

def mycallback(val ):
    print("Callback: "+str(val))


def myfunc1(n=3):
    print("start myfunc1")
    slist = range(n)
    pool = mp.Pool(5)
    [pool.apply_async(worker,args=(s,), callback=mycallback) for s in slist]
    pool.close()
    pool.join()

if __name__ == "__main__":
    myfunc1()



test2.py

from pythonProjectTEST.test1 import myfunc1
import multiprocessing as mp

def mycallback(val ):
    print("CallbackMaster: "+str(val))

if __name__ == "__main__":
    # This works
    #myfunc1(5)

    # This does not
    slist = range(6)
    pool = mp.Pool(3)
    [pool.apply_async(myfunc1,args=(s,), callback=mycallback) for s in slist]
    pool.close()
    pool.join()

Please include the error details. – noxdafox May 26 '21 at 17:04 — noxdafox, May 26 '21 at 17:04
OK, I added it to the question – ManInMoon May 26 '21 at 18:09 — ManInMoon, May 26 '21 at 18:09

score 0 · Answer 1 · answered Jun 10 '21 at 22:11

You are not allowed to spawn a daemon process from another daemon process. Note how the main of test2 spawns processes to call myfunc1 and then myfunc1 spawns processes to call worker. I suspect this restriction is to reduce the chances of fork bombs or deadlocks. If you really want to do this, there are workarounds: Python Process Pool non-daemonic?. However, I would avoid it if possible.

To debug an issue like this, it is often convenient to add an error callback. For example, the following code gives you a helpful error message "Error: daemonic processes are not allowed to have children":

def errorcallback(val):
    print("Error: %s" % str(val))
 ...
[pool.apply_async(myfunc1,args=(s,), callback=mycallback, error_callback=errorcallback ) for s in slist]

The method apply_async will normally eat errors unless you specify an error_calblack (see the documentation here). The arguments in square brackets are optional, but you can add them one by one with the specified names.

apply_async(func[, args[, kwds[, callback[, error_callback]]]])

"If error_callback is specified then it should be a callable which accepts a single argument. If the target function fails, then the error_callback is called with the exception instance."

Pool issue when called from parent pool Python

1 Answers1