Popen.communicate raises ValueError: 'I/O operation on closed file'

Question

I get the following error when I run my code.

    Uncaught exception <class 'ValueError'>: I/O operation on closed file.Traceback (most recent call last):
    File "/home/gidumah/miniconda/envs/ytune/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/gidumah/miniconda/envs/ytune/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/gidumah/ytopt/ytopt/ytopt/search/ambs.py", line 128, in <module>
        search.main()
      File "/home/gidumah/ytopt/ytopt/ytopt/search/ambs.py", line 105, in main
        results = list(self.evaluator.get_finished_evals())
      File "/home/gidumah/ytopt/ytopt/ytopt/evaluator/evaluate.py", line 200, in get_finished_evals
        y = future.result()
      File "/home/gidumah/ytopt/ytopt/ytopt/evaluator/subprocess_evaluator.py", line 41, in result
        stdout, stderr_data = self.proc.communicate()
      File "/home/gidumah/miniconda/envs/ytune/lib/python3.7/subprocess.py", line 951, in communicate
        stdout = self.stdout.read()
    ValueError: I/O operation on closed file.

The code I was running is subprocess_evaluator.py, which is given below:

class PopenFuture:
    FAIL_RETURN_VALUE = Evaluator.FAIL_RETURN_VALUE

    def __init__(self, args, parse_fxn):
        self.proc = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE,
                                     stderr=subprocess.STDOUT, encoding='utf-8')

        self._state = 'active'
        self._result = None
        self._parse = parse_fxn

    def _poll(self):
        if not self._state == 'active':
            return
        retcode = self.proc.poll()
        if retcode is None:
            self._state = 'active'
            stdout, stderr_data = self.proc.communicate()
            tmp_res = self._parse(stdout)
            if tmp_res != sys.float_info.max:
                self._result = tmp_res
        elif retcode == 0:
            self._state = 'done'
        else:
            self._state = 'failed'

    def result(self):
        if self._result is not None:
            return self._result
        self.proc.wait()
        stdout, stderr_data = self.proc.communicate()
        if self.done:
            self._result = self._parse(stdout)
        else:
            self._result = self.FAIL_RETURN_VALUE
            logger.error(f"Eval failed: {stdout}")
        ####
        # if stdout:
        #     print (stdout)#.split('\n')[:-2])#)[:-1])
        ####   
        return self._result

It says the error is in line 41, which is given as stdout, stderr_data = self.proc.communicate()

Is there something I am not doing right with the communicate call?

In the future, please paste code and tracebacks as plain text surrounded by \`\`\` code tags, instead of whatever IDE option which generates HTML formatting that I had to remove manually. — SuperStormer, Jul 01 '22 at 22:36

score 0 · Answer 1 · answered Jun 30 '22 at 23:56

Accoring to the documentation:

Popen.communicate(input=None, timeout=None)

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate and set the returncode attribute. The optional input argument should be data to be sent to the child process, or None, if no data should be sent to the child. If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.

Note that it waits for the process to terminate. Therefore Popen.communicate cannot be run more than once. The error in the context of Popen.communicate means that the program already terminated the process and closed the pipes, and raised the exception upon second communicate call.

It seems that you are using code of the ytopt library. Please check if you use the code correctly, note that it calls communicate() in few different places, sometimes with passing on ValueErrors but sometimes not. Also, check out Thomas Wouters solution to the problem:

If you want to use communicate, you have to either restart the process in the loop, or give it a single string that is all the input from the generator. If you want to do streaming communication, sending data bit by bit, then you have to not use communicate. Instead, you would need to write to p.stdin while reading from p.stdout and p.stderr. Doing this is tricky, because you can't tell which output is caused by which input, and because you can easily run into deadlocks. There are third-party libraries that can help you with this, like Twisted.

If you want to do this interactively, sending some data and then waiting for and processing the result before sending more data, things get even harder. You should probably use a third-party library like pexpect for that.

Of course, if you can get away with just starting the process inside the loop, that would be a lot easier:
cmd = ['intersectBed', '-a', 'stdin', '-b', bedfile]
for entry in my_entry_generator:
    p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    output = p.communicate(input='\t'.join(entry) + '\n')[0]
    print output

Popen.communicate raises ValueError: 'I/O operation on closed file'

1 Answers1