32

I need to do something like this post, but I need to create a subprocess that can be given input and give output many times. The accepted answer of that post has good code...

from subprocess import Popen, PIPE, STDOUT

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0]
print(grep_stdout.decode())

# four
# five

...that I would like to continue like this:

grep_stdout2 = p.communicate(input=b'spam\neggs\nfrench fries\nbacon\nspam\nspam\n')[0]
print(grep_stdout2.decode())

# french fries

But alas, I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 928, in communicate
    raise ValueError("Cannot send input after starting communication")
ValueError: Cannot send input after starting communication

The proc.stdin.write() method not enable you to collect output, if I understand correctly. What is the simplest way to keep the lines open for ongoing input/output?

Edit: ====================

It looks like pexpect is a useful library for what I am trying to do, but I am having trouble getting it to work. Here is a more complete explanation of my actual task. I am using hfst to get grammar analyses of individual (Russian) words. The following demonstrates its behavior in a bash shell:

$ hfst-lookup analyser-gt-desc.hfstol
> слово
слово   слово+N+Neu+Inan+Sg+Acc 0.000000
слово   слово+N+Neu+Inan+Sg+Nom 0.000000

> сработай
сработай    сработать+V+Perf+IV+Imp+Sg2 0.000000
сработай    сработать+V+Perf+TV+Imp+Sg2 0.000000

>

I want my script to be able to get the analyses of one form at a time. I tried code like this, but it is not working.

import pexpect

analyzer = pexpect.spawnu('hfst-lookup analyser-gt-desc.hfstol')
for newWord in ['слово','сработай'] :
    print('Trying', newWord, '...')
    analyzer.expect('> ')
    analyzer.sendline( newWord )
    print(analyzer.before)

# trying слово ...
# 
# trying сработай ...
# слово
# слово слово+N+Neu+Inan+Sg+Acc 0.000000
# слово слово+N+Neu+Inan+Sg+Nom 0.000000
# 
#

I obviously have misunderstood what pexpect.before does. How can I get the output for each word, one at a time?

Community
  • 1
  • 1
reynoldsnlp
  • 1,072
  • 1
  • 18
  • 45
  • "The proc.stdin.write() method not enable you to collect output, " You can still get output, you just have to get it from proc.stdout and proc.stderr. – Joshua Taylor Feb 19 '15 at 20:14
  • 3
    Is this windows or linux? On linux, the pexpect module is a good choice for subprocess interaction. – tdelaney Feb 19 '15 at 20:19
  • 1
    what are you trying to do? – Padraic Cunningham Feb 19 '15 at 20:45
  • 2
    mandatory reading if you want "multiple input and outputs": [Q: Why not just use a pipe (popen())?](http://pexpect.readthedocs.org/en/latest/FAQ.html#whynotpipe) – jfs Feb 20 '15 at 03:16
  • @PadraicCunningham I am trying to get a proof of concept example that I can expand to use in a script that will need to interact with my process 1,000,000+ times. My current script uses subprocess.check_output, which is much too slow, since it has to initiate the process for each interaction. – reynoldsnlp Feb 20 '15 at 07:55
  • @tdelaney This is on OS X. – reynoldsnlp Feb 20 '15 at 10:57
  • 2
    the sequence should be: 0. wait for the first prompt 1. send word 2. wait for the prompt, get response (`.after`?) for the word 3. repeat 1-2 – jfs Feb 20 '15 at 16:10
  • сработай :D сработай ;[ – 0dminnimda Mar 11 '22 at 17:24

4 Answers4

33

Popen.communicate() is a helper method that does a one-time write of data to stdin and creates threads to pull data from stdout and stderr. It closes stdin when its done writing data and reads stdout and stderr until those pipes close. You can't do a second communicate because the child has already exited by the time it returns.

An interactive session with a child process is quite a bit more complicated.

One problem is whether the child process even recognizes that it should be interactive. In the C libraries that most command line programs use for interaction, programs run from terminals (e.g., a linux console or "pty" pseudo-terminal) are interactive and flush their output frequently, but those run from other programs via PIPES are non-interactive and flush their output infrequently.

Another is how you should read and process stdout and stderr without deadlocking. For instance, if you block reading stdout, but stderr fills its pipe, the child will halt and you are stuck. You can use threads to pull both into internal buffers.

Yet another is how you deal with a child that exits unexpectedly.

For "unixy" systems like linux and OSX, the pexpect module is written to handle the complexities of an interactive child process. For Windows, there is no good tool that I know of to do it.

tdelaney
  • 73,364
  • 6
  • 83
  • 116
  • 1
    you can deadlock even with only stdin/stdout e.g., how do you know when to read from grep's stdout after you've written something? Also, the child process may bypass stdin/stdout completely (typical example: a password prompt). See [the link in the comment above](http://stackoverflow.com/questions/28616018/multiple-inputs-and-outputs-in-python-subprocess-communicate#comment45544447_28616018). In many cases these issues could be resolved using threads, fcntl, async.io: select/poll/epoll/kqueue/iocp and/or pty. And [sometimes it is enough to be careful](http://stackoverflow.com/q/7897202/4279) – jfs Feb 20 '15 at 03:36
  • @J.F.Sebastian - and then there are programs that read the terminal type to colorize or full-screen output. It can be a challenge. – tdelaney Feb 20 '15 at 16:23
  • if a program does not provide options to override such behavior (such as `--color`); it can be considered a bug. The default behavior should be for an interactive user – less typing, concise output. Full screen shouldn't be used unless it is absolutely necessary. – jfs Feb 20 '15 at 16:35
  • Note to Alice Carvalho who edited my post - you have an interesting contribution dealing with pexpect updates, but its not part of my answer. So, a comment or alternate answer would both be good. – tdelaney Jul 25 '23 at 21:53
20

This answer should be attributed to @J.F.Sebastian. Thanks for the comments!

The following code got my expected behavior:

import pexpect

analyzer = pexpect.spawn('hfst-lookup analyser-gt-desc.hfstol', encoding='utf-8')
analyzer.expect('> ')

for word in ['слово', 'сработай']:
    print('Trying', word, '...')
    analyzer.sendline(word)
    analyzer.expect('> ')
    print(analyzer.before)
reynoldsnlp
  • 1,072
  • 1
  • 18
  • 45
  • `AttributeError: module 'pexpect' has no attribute 'spawnu'` – Mooncrater Jul 28 '20 at 08:14
  • 1
    @Mooncrater it looks like `pexpect.spawnu` was deprecated in favor of using `spawn(encoding='utf-8')`. I updated the answer accordingly. However, it is still in the source code (https://github.com/pexpect/pexpect/blob/master/pexpect/pty_spawn.py#L857-L860), so I wonder if you've installed `pexpect` correctly. – reynoldsnlp Jul 28 '20 at 13:45
  • Expect if good to work to simulate interaction with a terminal, for programs which will change their behavior when they are in a pipe line. However, it's not good if you communicate with binary data. – Camion Jul 15 '21 at 05:20
  • @Camion for binary data, just omit the `encoding='utf-8'` argument. By default, `spawn` expects binary. – reynoldsnlp Jul 15 '21 at 17:12
  • I don't think it's that simple because of the new lines processing. I didn't check it right now, but I believe even without the encoding pexpect will convert the end of lines characters to windows like end of line (`\r\n`), and you might also have problems if the end of you data is not a new line. – Camion Jul 15 '21 at 19:21
9

Whenever you want to send input to the process, use proc.stdin.write(). Whenever you want to get output from the process, use proc.stdout.read(). Both stdin and stdout arguments to the constructor need to be set to PIPE.

Tom Hunt
  • 916
  • 7
  • 21
  • 1
    It works perfectly. The proc takes one input and send out one output. – dom free Mar 23 '18 at 07:52
  • 3
    This should be used with caution with line splitters and flushing the PIPE as it may create deadlocks. More could be reached in this great blog post: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/ – Maged Saeed May 14 '20 at 03:02
2

HFST has Python bindings: https://pypi.python.org/pypi/hfst

Using those should avoid the whole flushing issue, and will give you a cleaner API to work with than parsing the string output from pexpect.

From the Python REPL, you can get some doc's on the bindings with

dir(hfst)
help(hfst.HfstTransducer)

or read https://hfst.github.io/python/3.12.2/QuickStart.html

Snatching the relevant parts of the docs:

istr = hfst.HfstInputStream('hfst-lookup analyser-gt-desc.hfstol')
transducers = []
while not (istr.is_eof()):
    transducers.append(istr.read())
istr.close()
print("Read %i transducers in total." % len(transducers))
if len(transducers) == 1:
  out = transducers[0].lookup_optimize("слово")
  print("got %s" % (out,))
else: 
  pass # or handle >1 fst in the file, though I'm guessing you don't use that feature
unhammer
  • 4,306
  • 2
  • 39
  • 52