2

I'm trying to read from sys.stdin from inside of a Python Process object, but I keep getting a "ValueError: I/O operation on closed file" result. Here's a quick example:

import sys
from multiprocessing import Process

def do_something(input_data):
    for x in input_data:
        print x


input=sys.stdin

p = Process(target=do_something, args=(input,))
p.start() 
p.join() #Wait for Process to complete

The above script always fails with:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "example.py", line 6, in do_something
    for x in input_data:
ValueError: I/O operation on closed file

Just calling do_something(input) works fine without using a Process, of course. Creating a Pipe() object seems to help - I can write the contents of stdin to the Pipe and get the results in string form from within the process - but I actually need the input to file-like form for some downstream operations. I could dump the contents to a file and the re-read it in from within the Process, but that seems pretty clumsy, especially if the stdin is really big. Is there some easy way to read from sys.stdin from within a Process?

homesalad
  • 477
  • 1
  • 5
  • 10

2 Answers2

4

This is because before the Process is started, stdin is closed. Otherwise it could happen that both the parent and child process (or multiple child processes) try to read from the same stdin, which is a bad idea.

In the child process sys.stdin is actually redirected to /dev/null:

from multiprocessing import Process
import sys

def test(*args):
    print(args)
    print(sys.stdin, sys.stdin.fileno())

if __name__ == '__main__':
    p = Process(target=test, args=(sys.stdin,))
    p.start()
    p.join()

should print something similar to this:

(<closed file '<stdin>', mode 'r' at 0x7f3b4564b0c0>,)
(<open file '/dev/null', mode 'r' at 0x7f3b43a9e0c0>, 3)

The passed argument here is a reference to a closed file object, trying to use it will raise the error you've seen.

You could get around this by using os.dup() on sys.stdin.fileno() in the parent and pass the returned copy of the file descriptor to the child as argument, where you can then use os.fdopen() to work with it.

The cleaner solution would probably be to read the input in the parent process and pass it to the child using a multiprocessing.Queue.

mata
  • 67,110
  • 10
  • 163
  • 162
0

You must be closing a file that you are trying to write to at some point. Check your code and try removing all lines that close files (fileVariableName.close()) and see if it works. If it does then re-add them one by one to find the problem. Once you have found the line causing the issues try moving it further into the program (call it later) and see if that fixes your problems.

EDIT: change 

def do_something(input_data):
    for x in input_data:
        print x

to

def do_something():
    for x in sys.stdin:
        print x

and get rid of input = sys.stdin

NendoTaka
  • 1,224
  • 8
  • 14
  • The example code I posted demonstrates the issue, but it doesn't include any explicit file closings, so I'm pretty sure that's not it. Also remember that the "file" is sys.stdin, not a regular file. – homesalad Jun 02 '15 at 11:52
  • try changing `input=sys.stdin` to `input=sys.stdin.readlines()` – NendoTaka Jun 02 '15 at 13:03
  • That helps, but it reads all of stdin into memory before continuing, which is the behavior I'm hoping to avoid. The input may be very large, and I'd like to not read all of it before beginning the Process – homesalad Jun 03 '15 at 01:22
  • Well what is happening is that you are making input equal to `sys.stdin` itself not the value that is there (kind of like a file) but then when you are calling it later in your function it is not open for certain reasons why not just try a quick change. I will edit my answer because there is not enough space here. – NendoTaka Jun 03 '15 at 13:52