2

Answering this question about nested pools for multiprocessing:

https://stackoverflow.com/a/40852258/6522112

The code I proposed was not responding the way I wanted. Which is to say if ThreadingPool(8) and ProcessingPool(3), I expected the number of processes to be 24 but I only got 3! So I moved on using this https://stackoverflow.com/a/8963618/6522112 which allows nested process pool by setting the daemon flag to False. This works perfectly but:

  1. Was it normal to only get 3 processes instead of 24?
  2. Am I doing it correctly with this hack?
  3. The final aim would be to do this using several nodes on a cluster. Does this prevent it?
  4. I am not sure about the type of map to consider but it seems that only the imap combination is doing the job. Any insight?

Here is the modified code that uses pathos from @MikeMcKerns instead of simple multiprocessing:

import pathos
import multiprocess


class NoDaemonProcess(multiprocess.Process):

    """NoDaemonProcess class.

    Inherit from :class:`multiprocessing.Process`.
    ``daemon`` attribute always returns False.

    """

    def _get_daemon(self):
        return False

    def _set_daemon(self, value):
        pass

    daemon = property(_get_daemon, _set_daemon)


class NestedPool(pathos.multiprocessing.Pool):

    """NestedPool class.

    Inherit from :class:`pathos.multiprocessing.Pool`.
    Enable nested process pool.

    """

    Process = NoDaemonProcess

Care to be taken if you want to use this, you have to terminate the inner pools otherwise you will end up with zombies:

import NestedPool

def triple(x):
    return 3*x

def refork(x):
    pool = NestedPool(3)
    result = pool.imap(triple, range(5))
    result = list(result)
    pool.terminate()
    return result

pool = NestedPool(8)
result = pool.imap(refork, range(3))
Community
  • 1
  • 1
tupui
  • 5,738
  • 3
  • 31
  • 52

0 Answers0