36

It's not the first time I'm having this problem, and it's really bugging me. Whenever I open a pipe using the Python subprocess module, I can only communicate with it once, as the documentation specifies: Read data from stdout and stderr, until end-of-file is reached

proc = sub.Popen("psql -h darwin -d main_db".split(),stdin=sub.PIPE,stdout=sub.PIPE)
print proc.communicate("select a,b,result from experiment_1412;\n")[0]
print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]

The problem here is that the second time, Python isn't happy. Indeed, he decided to close the file after the first communicate:

Traceback (most recent call last):
File "a.py", line 30, in <module>
    print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]
File "/usr/lib64/python2.5/subprocess.py", line 667, in communicate
    return self._communicate(input)
File "/usr/lib64/python2.5/subprocess.py", line 1124, in _communicate
     self.stdin.flush()
ValueError: I/O operation on closed file

Are multiple communications allowed?

Manux
  • 3,643
  • 4
  • 30
  • 42

4 Answers4

26

I think you misunderstand communicate...

http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

communicate sends a string to the other process and then waits on it to finish... (Like you said waits for the EOF listening to the stdout & stderror)

What you should do instead is:

proc.stdin.write('message')

# ...figure out how long or why you need to wait...

proc.stdin.write('message2')

(and if you need to get the stdout or stderr you'd use proc.stdout or proc.stderr)

Terence Honles
  • 834
  • 6
  • 9
  • 1
    I realized you might want to look @ http://stackoverflow.com/questions/375427/non-blocking-read-on-a-stream-in-python (It's a very similar problem) – Terence Honles Jun 18 '10 at 01:53
  • 1
    Yes indeed, I needed non-blocking reads, because the write might always depend on what was read. – Manux Jun 18 '10 at 02:37
  • 1
    Warning from the [`subprocess` documentation](https://docs.python.org/3/library/subprocess.html#subprocess.Popen.stderr): > Warning: Use `communicate()` rather than `.stdin.write`, `.stdout.read` or `.stderr.read` to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process. – paime Mar 18 '21 at 13:41
6

I've had this problem before, and as far as I could ever figure, you couldn't do this with subprocess (which, I agree, is very counterintuitive if true). I ended up using pexpect (obtainable from PyPI).

gilesc
  • 1,969
  • 1
  • 14
  • 16
3

You can use:

proc.stdin.write('input')    
if proc.stdout.closed:
    print(proc.stdout)
Daniel
  • 39
  • 2
3

You can do this simply with single call of communicate():

query1 = 'select a,b,result from experiment_1412;'
query1 = 'select theta,zeta,result from experiment_2099;'
concat_query = "{}\n{}".format(query1, query2)
print(proc.communicate(input=concat_query.encode('utf-8'))[0])

The key-point here is that you only write once to stdin, and \n serve as EOL. your psql subprocess reads from stdin until \n, then after it finishes the first query, it goes to stdin again, by which time only the second query string is left in the buffer.

zimplex
  • 31
  • 3
  • This buffers the entire input in memory, making the solution less than ideal for situations where streaming I/O is desired. – weaver Sep 30 '17 at 16:25