2

Using python 3.6, I have a problem like so: (1) there's a joblib.Parallel loop over embarrassingly parallel jobs (2) the jobs themselves are fairly time intensive c++ native objects that occasionally segfault and whose code I cannot modify.

To guard against the segfaults, I attempted to wrap the jobs themselves inside a multiprocessing Process. Unfortunately, python itself throws an assertion error with daemonic processes are not allowed to have children with this solution.

So I took the solution posted here and tried inheriting from Process: https://stackoverflow.com/a/8963618/614684

That didn't work either, and so I came up with the following solution which DOES work:

class NoDaemonProcess(multiprocessing.Process):
  def __init__(self, group=None, target=None, name=None, args=(), kwargs={},
               *, daemon=None):
    super(NoDaemonProcess, self).__init__(group, target, name, args, kwargs, daemon=daemon)
    if 'daemon' in multiprocessing.process._current_process._config:
      del multiprocessing.process._current_process._config['daemon']
    self._config = multiprocessing.process._current_process._config.copy()
  # make 'daemon' attribute always return False                                                                                                                                              
  def _get_daemon(self):
    return False
  def _set_daemon(self, value):
    pass
  daemon = property(_get_daemon, _set_daemon)

Basically, I modify the global state of the multiprocessing package to delete the fact that the current process is a daemon.

Is there a better way to do this? I would appreciate any help in making this more robust and reliable.

JasonMond
  • 1,440
  • 1
  • 16
  • 30

1 Answers1

-1

Keep It Simple, Stupid :)

Your error message is at https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py#L110 .

You can use a regular subprocess as part of your worker payload. It won't be subject to any multiprocessing logic. To avoid creating a separate Python program specifically for that, just do

subprocess.Popen([sys.executable,__file__,"<arg>"])

and make <arg> trigger the necessary logic in the if __name__ == '__main__' block.

To get the result from the subprocess, either print it to stdout in any machine-readable format, or use any IPC mechanism.

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
  • I'm aware of the source of the error message. I arrived at my solution by looking at the multiprocessing source. And subprocess won't work. The job is with c++ library whose access is provided through cython. It's not a cmdline call. – JasonMond Jun 11 '18 at 02:05
  • @JasonMond You can launch the same Python file with a command line argument that will trigger the necessary logic in the `if __name__ == '__main__'` block. – ivan_pozdeev Jun 11 '18 at 03:34
  • that won't help either. Among many arguments to the job is a list of xml objects. Which can be in the order of 100s of MBs. Outside of writing those files to a file system where space is at a premium (and I have 1000s of these jobs running in parallel), I don't see how those things can be passed as arguments through subprocess in a clean manner. And then the return value is a structured python object created in cython. I could probably create a json based serialization and deserialization function so that it can go through subprocess, but that's also a decent amount of work. – JasonMond Jun 11 '18 at 14:06
  • @JasonMond just the same, stdin or IPC. `multiprocessing` uses `pickle` under the hood, so it's as simple as `pickle.dump(args,process.stdin)` and `.load(sys.stdin)` on the other end (`sys.stdin.buffer` in Py3 to read in binary mode). – ivan_pozdeev Jun 11 '18 at 17:09
  • @JasonMond and if you use fork, you don't need to send input data at all, only get the result back. – ivan_pozdeev Jun 11 '18 at 17:32