0

I currently have a worker subprocess that does a lot of processing for my main application. I have my stdin and stdout already connected between the two, but now I need more than just these two between the main application and the subprocess worker, since the subprocess worker is heavily threaded and should be able to run multiple different workload at the same time.

So for each workload, I want to dynamically create a separate pipe between the main and subprocess. I don't want to run more than 1 subprocess worker in the background, and want everything to run on a single subprocess.

My problem that I ran into is creating named pipes between the main and subprocess, that will work on both Unix an Windows. Unix has a os.mkfifo() that can be used with tempfiles, but that does not work on Windows. The os.pipe() will not work, because the memory block being allocated is only for the main application, and I have no way to link that with the subprocess.

So basically, 

tmp_read, tmp_write = os.pipe( )

for both tmp_read and tmp_write, they are represented as integers, or memory blocks on the main app stack. I can't send these integers to the subprocess and connect, as the subprocess has no idea what they mean. Am I missing something, or is it not possible to share an undefined number of pipes between the processes using IPC? I can't use sockets for IPC either, since the computers that this has to run on are heavily restricted and I don't want to deal with blocked ports.

Hugo Coppejans
  • 46
  • 6
  • Related, possibly duplicate: https://stackoverflow.com/questions/2989823/how-to-pass-file-descriptors-from-parent-to-child-in-python – Michał Politowski Jan 08 '19 at 10:33
  • With subprocess, you can inherit inheritable handles by passing `close_fds=False`. Get the handle value, e.g. `htmp_read = msvcrt.get_osfhandle(tmp_read)`). Make it inheritable; it's easy in Python 3, e.g. `os.set_handle_inheritable(htmp_read, True`). Send the integer handle values as strings on the command line, or an environment variable, or via the child's stdin. After the child process is created, close the parent's copy, e.g. `os.close(tmp_read)`. In the child, create a file descriptor for the integer handle value, e.g. `fd = msvcrt.open_osfhandle(h, os.O_RDONLY)`. – Eryk Sun Jan 08 '19 at 10:53

0 Answers0