Python subprocess - write multiple stdin

Question

I need to open an R script and supply it with input formulated by a separate python script. The subprocess module seems to be a good way to do this.

I have encountered some puzzling results though, namely that I can apparently write once and only once via p.stdin. Here is what I have so far:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['r --no-save'],stdin=PIPE,stdout=PIPE,stderr=PIPE,shell=True)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput')

What happens when I run this code is that the first instance of stdin.write() performs as expected (and opens my R script), but the second line does nothing, and the subprocess (really, the R script) exits with an error, indicating that the subprocessed received no input where input was expected and therefore terminated.

N.B. - In a perfect world, I would just interact directly through R, but this particular script requires complex inputs that cannot be entered directly for practical purposes. Also, rpy / rpy2 is not an option, because end-users of this script will not necessarily have access to that module or its dependencies. rscript is also not an option (for many reasons, but mainly because of variability in the end-users R configurations).

Finally, p.communicate is not an option, because apparently that will close the process after writing and I need to keep it open.

Thanks in advance

good question, and no, including `p.stdin.flush()` between the write commands does does not change the result. — J R, Apr 03 '13 at 14:57
According to the subprocess module documentation for Popen... Note Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer. Maybe something wacky is going on in the script? Have you tried just issuing 2 simple commands like, p.stdin.write("a = 7") p.stdin.write("print(a)") instead of the script? — kith, Apr 03 '13 at 15:01
That was a helpful suggestion, thanks. It does indeed appear to be something internal to my script, because when I issue 2 separate commands, as you suggest, I get the expected behavior from `p.stdin.write()`. I will troubleshoot the R script - many thanks! — J R, Apr 03 '13 at 17:13
Can you tell us how you access stdin in the R script? "/dev/stdin" or "stdin"? With the former one I experienced problems — vodka, Apr 03 '13 at 17:42
Hmm, I'm afraid I don't know the answer to that question. I can tell you though, that my R script does not explicitly call on either `/dev/stdin` or `stdin` — J R, Apr 03 '13 at 17:48
Uhm, ok. Do you have access to this script? At least the relevant parts where the standard input is read? To see why I asked check the notes at the end of help(stdin) — vodka, Apr 03 '13 at 18:09
if the child R script reads directly from a terminal, you could try `pexpect`, `pty` modules to send/receive such input/output. Related: [Capture “Segmentation fault” message for a crashed subprocess: no out and err after a call to communicate()](http://stackoverflow.com/q/22250893/4279) — jfs, Apr 11 '14 at 22:02

Hai Vu · Answer 1 · 2014-04-11T22:26:20.173

What you need is to call .communicate():

from subprocess import Popen, PIPE, STDOUT

p = Popen(
    ['r', '--nosave'],
    stdin=PIPE,
    stdout=PIPE,
    stderr=PIPE)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput\n')
p.stdin.write('q\n')

stdout, stderr = p.communicate()

print '---STDOUT---'
print stdout
print '---STDERR---'
print stderr
print '---'

Discussion

I don't use the shell=True and it seems working with my fake R script since I don't have R install in my system. You might or might not need it.
I prefer breaking the command line up into a list of string as shown, but a single string such as r --nosave will work as well; just don't do them both at the same time.
Don't forget that stdin.write() does not write the new line character \n, you have to supply that yourself.

Update

My first attempt was off the mark, I hope this second attempt gets closer. As J.F. Sebastian suggested, you might want to use pexpect:

import pexpect
import sys

if __name__ == '__main__':
    prompt = '> ' # Don't know what the R prompt looks like
    lines = ['one', 'two', 'three']

    r = pexpect.spawn('r --no-save', logfile=sys.stdout)
    for line in lines:
        r.expect(prompt)
        r.sendline(line)

    # If you want to interact with your script, use these two lines
    # Otherwise, comment them out
    r.logfile = None # Turn off logging to sys.stdout
    r.interact()

Discussion

You might need to install pexpect. I did it with pip install pexpect
If you don't want to interact with the system, comment out the last two line, but make sure to send some signal for the R script to exit.
spawn() returns a spawn object, see doc here.

OP already said that `p.communicate()` is not suitable because it waits for the process to complete. Also use `p.communicate("\n".join(["source('myrscript.R')", "myfirstinput", "q"]))` instead of `p.stdin.write()` calls otherwise you may cause a deadlock — jfs, Apr 11 '14 at 22:00

Python subprocess - write multiple stdin

1 Answers1

Discussion

Update

Discussion