Using stdin in a child Process

Question

So I have a program, in the "main" process I fire off a new Process object which (what I want) is to read lines from stdin and append them to a Queue object.

Essentially the basic system setup is that there is a "command getting" process which the user will enter commands/queries, and I need to get those queries to other subsystems running in separate processes. My thinking is to share these via a multiprocessing.Queue which the other systems can read from.

What I have (focusing on just the getting the commands/queries) is basically:

def sub_proc(q):
    some_str = ""
    while True:
        some_str = raw_input("> ")
        if some_str.lower() == "quit":
            return
        q.put_nowait(some_str)

if __name__ == "__main__":
    q = Queue()
    qproc = Process(target=sub_proc, args=(q,))
    qproc.start()
    qproc.join()

    # now at this point q should contain all the strings entered by the user

The problem is that I get:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/blah/blah/blah/blah.py", line 325, in sub_proc
    some_str = raw_input("> ")
  File "/randompathhere/eclipse/plugins/org.python.pydev_2.1.0.2011052613/PySrc/pydev_sitecustomize/sitecustomize.py", line 181, in raw_input
    ret = original_raw_input(prompt)
EOFError: EOF when reading a line

How do?

Answer 1

I solved a similar issue by passing the original stdin file descriptor to the child process and re-opening it there.

def sub_proc(q,fileno):
    sys.stdin = os.fdopen(fileno)  #open stdin in this process
    some_str = ""
    while True:
        some_str = raw_input("> ")

        if some_str.lower() == "quit":
            return
        q.put_nowait(some_str)

if __name__ == "__main__":
    q = Queue()
    fn = sys.stdin.fileno() #get original file descriptor
    qproc = Process(target=sub_proc, args=(q,fn))
    qproc.start()
    qproc.join()

This worked for my relatively simple case. I was even able to use the readline module on the re-opened stream. I don't know how robust it is for more complex systems.

Answer 2

In short, the main process and your second process don't share the same STDIN.

from multiprocessing import Process, Queue
import sys

def sub_proc():
    print sys.stdin.fileno()

if __name__ == "__main__":
    print sys.stdin.fileno()
    qproc = Process(target=sub_proc)
    qproc.start()
    qproc.join()

Run that and you should get two different results for sys.stdin.fileno()

Unfortunately, that doesn't solve your problem. What are you trying to do?

Answer 3

If you don't want to pass stdin to the target processes function, like in @Ashelly's answer, or just need to do it for many different processes, you can do it with multiprocessing.Pool via the initializer argument:

import os, sys, multiprocessing

def square(num=None):
    if not num:
        num = int(raw_input('square what? ')) 
    return num ** 2

def initialize(fd):
    sys.stdin = os.fdopen(fd)

initargs = [sys.stdin.fileno()]
pool = multiprocessing.Pool(initializer=initialize, initargs=initargs)
pool.apply(square, [3])
pool.apply(square)

the above example will print the number 9, followed by a prompt for input and then the square of the input number.

Just be careful not to have multiple child processes reading from the same descriptor at the same time or things may get... confusing.

Answer 4

You could use threading and keep it all on the same process:

from multiprocessing import Queue
from Queue import Empty
from threading import Thread

def sub_proc(q):
    some_str = ""
    while True:
        some_str = raw_input("> ")
        if some_str.lower() == "quit":
            return
        q.put_nowait(some_str)

if __name__ == "__main__":
    q = Queue()
    qproc = Thread(target=sub_proc, args=(q,))
    qproc.start()
    qproc.join()

    while True:
        try:
            print q.get(False)
        except Empty:
            break