1

Im trying to stdout.readline and put the results (i.e each line, at the time of printing them to the terminal) on a multiprocessing.Queue for us in another .py file. However, the call:

res = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1 )
with res.stdout:
    for line in iter(res.stdout.readline, b''):
        print line
res.wait()

Will block and the results will be printed after the process is complete (or not at all if exit code isn't returned).

I've browsed SO for answers to this, and tried setting bufsize=1, spawning threads that handle the reading, using filedescriptors, etc. None seem to work. I might have to use the module pexpect but I'm not sure how it works yet.

I have also tried

 def enqueue_output(self, out, queue):
    for line in iter(out.readline, b''):
        queue.put([line])
    out.close()

To put the data on the queue, but since out.readline seems to block, the result will be the same.

In short: How do I make the subprocess output available to me at the time of print? It prints chunks of 1-10 lines at a time, however these are returned to me when the process completes, separated by newlines as well..

Related:

Python subprocess readlines() hangs

Python: read streaming input from subprocess.communicate()

Non-blocking read on a subprocess.PIPE in python

Community
  • 1
  • 1
enrm
  • 645
  • 1
  • 8
  • 22
  • You need to find out whether the program has a command-line option to disable buffering or to use an interactive mode, like Python's "-u" and "-i" options. Otherwise it most likely buffers writing to its `stdout` when it sees that it's a pipe. Output is only written to the pipe when the buffer is full, or when the program manually flushes the buffer, or when it exits. Python, on the other end of the pipe, has no control over this behavior. – Eryk Sun Jun 20 '16 at 10:47
  • You mention pexpect and have the question tagged "pty", but Windows doesn't have terminals and pseudoterminals. The closest to pexpect is winpexpect, but that uses pipes, so it won't solve the buffering problem. Handling this properly in Windows requires the console API, maybe by attaching a helper program to a windowless console to read the program's output and write to its input. I haven't seen a Python module that implements this for Windows. – Eryk Sun Jun 20 '16 at 10:53
  • @eryksun I removd the pty- tag. I was afraid that the problem lies in the process I'm calling.. Will have to ask the responsible dev for that. There seems to be some issues with buffering whenrunning Windows and using the subprocess and commandline facilities.. – enrm Jun 20 '16 at 10:58
  • The same issues apply to Unix, but there are alternatives there, such as using pexpect with a pty. In Linux you can also run a program using `stdbuf`, which allows overriding the automatic buffering of C standard I/O (but not if the program specifically sets the buffering). It's based on the `LD_PRELOAD` environment variable. There's nothing similar in Windows. You could do DLL injection, but then the problem is the multitude of C runtime libraries in use and the lack of a global symbol table. – Eryk Sun Jun 20 '16 at 11:40
  • @eryksun thanks for the clarifications. Yeah, in unix, atleast you have some tools to combat the problem.. I don't know what to do in my position. Might have to fake some text for the users' sake.. – enrm Jun 20 '16 at 12:17
  • @eryksun apparently, the prints coming from the application are "printf"s.. Does that help? – enrm Jun 20 '16 at 12:36
  • No, it doesn't really help other than to confirm that the stream buffering is probably automatically set by the CRT (C runtime library). [`printf`](https://msdn.microsoft.com/en-us/library/wc7014hz.aspx) is a C standard I/O function that combines string formatting and writing to the `stdout` `FILE` stream. It's the stream itself that has buffering set, e.g. via [`setvbuf`](https://msdn.microsoft.com/en-us/library/86cebhfs.aspx). – Eryk Sun Jun 20 '16 at 12:47

1 Answers1

1

As explained by @eryksun, and confirmed by your comment, the cause of the buffering is the use of printf by the C application.

By default, printf buffers its output, but the output is flushed on newline or if a read occurs when the output is directed to a terminal. When the output is directed to a file or a pipe, the actual output only occurs when the buffer is full.

Fortunately on Windows, there is no low level buffering (*). That means that calling setvbuf(stdout, NULL, _IONBF, 0); near the beginning of the program would be enough. But unfortunately, you need no buffering at all (_IONBF), because line buffering on Windows is implemented as full buffering.

(*) On Unix or Linux systems, the underlying system call can add its own buffering. That means that a program using low level write(1, buf, strlen(buf)); will be unbuffered on Windows, but will still be buffered on Linux when standard output is connected to a pipe or a file.

Serge Ballesta
  • 143,923
  • 11
  • 122
  • 252
  • Upvoting for providing relevant information. So to answer the initial question: There is no way to retrieve the buffer until the stream is closed ? – enrm Jun 20 '16 at 13:58
  • More exactly, you will get full buffers from time to time until the stream is closed. But without changing the called program I really cannot imagine a way to avoid that buffering. – Serge Ballesta Jun 20 '16 at 14:02
  • Do you have a reference for low-level buffering of pipes on Linux or other Unix OS? The pipe itself has a buffer, which determines its capacity before a `write` will block or fail, but a `write` on one end should be immediately available for a `read` on the other end. With Windows I/O, general file access uses the system cache, and you need to open the file with the flag `FILE_FLAG_WRITE_THROUGH` to ensure data is immediately written (but some devices may lie to the kernel when it comes to their own caches); however, for pipes this only matters for a pipe that's accessed over the network. – Eryk Sun Jun 20 '16 at 14:27
  • @eryksun: I have no reference, I just did a test on FreeBSD using plain `write(1, buf, strlen(buf))` on sender, and reader was blocked. – Serge Ballesta Jun 20 '16 at 14:58
  • What did you do on the `read` end? If you try to `read` N bytes from a pipe, it may actually read less; you get however much is in the pipe. But it should only block if the pipe is empty and in blocking mode. – Eryk Sun Jun 20 '16 at 15:45
  • @eryksun: I used a mere `fgets`. And it was blocked so I assumed it could not read till the first newline. – Serge Ballesta Jun 20 '16 at 15:50
  • `fgets` is C standard I/O, reading from a `FILE` stream. It blocks until it reads up to EOF, a newline, or the number of bytes requested, less one for the terminating null. That it blocked has nothing to do with low-level buffering in the OS pipe. – Eryk Sun Jun 20 '16 at 16:16
  • @eryksun: I forgot to say that I was sending a newline every 60 bytes, and that the sender waited 1 second after each line. But after 1 minute (60 lines) the fgets had not returned... – Serge Ballesta Jun 20 '16 at 17:53
  • I can't reproduce this behavior in Linux, even after running the reading process with `stdbuf -i 4K` or calling `setvbuf` to force full buffering on `stdin`. But I don't want to analyze C standard I/O `fgets` between Linux and BSD. Surely if you use a raw `write` to one end of the pipe, then you should be able to `read` it immediately on the other end. – Eryk Sun Jun 20 '16 at 18:50