13

I am trying to employ a Subprocess in Python for keeping an external script open in a Server-like fashion. The external script first loads a model. Once this is done, it accepts requests via STDIN and returns processed strings to STDOUT.

So far, I've tried

tokenizer = subprocess.Popen([tokenizer_path, '-l', lang_prefix], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

However, I cannot use

tokenizer.stdin.write(input_string+'\n')
out = self._tokenizer.stdout.readline()

in order to repeatedly process input_strings by means of the subprocess – out will just be empty, no matter if I use stdout.read() or stdout.readline(). However, it works when I close the stdin with tokenizer.stdin.close() before reading STDOUT, but this closes the subprocess, which is not what I want as I would have to reload the whole external script again before sending another request.

Is there any way to use a subprocess in a server-like fashion in python without closing and re-opening it?

sam
  • 1,406
  • 2
  • 15
  • 25
  • Try flushing the stdin after doing a write, using `tokenizer.stdin.flush()` – onon15 Nov 28 '12 at 10:57
  • I have tried that already – without success, unfortunately... – sam Nov 28 '12 at 11:00
  • You might need an extra thread to get this to work. Check the implementation of `subprocess.Popen.communicate` to see how that's done. – Fred Foo Nov 28 '12 at 11:25
  • I did something similar where I had a deamon communicating with my main program via named pipes. It works well. Would you consider using named pipes rather? I will be happy to share my implementation – Sheena Nov 28 '12 at 11:27
  • @larsmans: I checked the implementation of .communicate() before; could you give me a hint what you mean by using «an extra thread»? – sam Nov 28 '12 at 11:32
  • @Sheena: that could be very helpful, indeed. Could you maybe post it on pastebin.com or something similar? – sam Nov 28 '12 at 11:36

2 Answers2

7

Thanks to this Answer, I found out that a slave handle must be used in order to properly communicate with the subprocess:

master, slave = pty.openpty()
tokenizer = subprocess.Popen(script, shell=True stdin=subprocess.PIPE, stdout=slave)
stdin_handle = process.stdin
stdout_handle = os.fdopen(master)

Now, I can communicate to the subprocess without closing it via

stdin_handle.write(input)
stdout_handle.readline() #gets the processed input
Community
  • 1
  • 1
sam
  • 1,406
  • 2
  • 15
  • 25
2

Your external script probably buffers its output, so you only can read it in the father when the buffer in the child is flushed (which the child must do itself). One way to make it flush its buffers is probably closing the input because then it terminates in a proper fashion and flushes its buffers in the process.

If you have control over the external program (i. e. if you can patch it), insert a flushing after the output is produced.

Otherwise programs sometimes can be made to not buffer their output by attaching them to a pseudo-TTY (many programs, including the stdlib, assume that when their output is going to a TTY, no buffering is wished). But this is a bit tricky.

Alfe
  • 56,346
  • 20
  • 107
  • 159
  • I actually just found a solution using Python's [Pseudo-terminal utilities](http://docs.python.org/2/library/pty.html), thanks to [this](http://stackoverflow.com/a/5413588/533362) answer. – sam Nov 28 '12 at 13:04
  • I still would prefer an explicit flushing of the co-processes buffers. – Alfe Nov 28 '12 at 13:21