python-Can we use tempfile with subprocess to get non buffering live output in python app

Question

I am trying to run one python file from python windows application.For that I have used subprocess.For getting live streaming output on app console I have tried the below statements.

With PIPE

p = subprocess.Popen(cmd,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT, shell=True)

for line in iter(p.stdout.readline, ''):
    print line

(or)

process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
    out = process.stdout.read(1)
    if out == '' and process.poll() != None:
        break
    if out != '':
        sys.stdout.write(out)
        sys.stdout.flush()

Not only above code ,tried so many methods.Getting same results like below:

1.Python windows app taking so much of time to run

2.Then the app window went to "not responding" state for long time

3.Then whole output is printed on the console

I know that the buffer overflow is happening in python app thats why i am not getting live output.

I posted so many queries for this still not getting solution.

Just now found and tried tempfile for this.But i am not sure this will give live streaming output.

Shall I try this way?

import tempfile
import subprocess

w = tempfile.NamedTemporaryFile()
p = subprocess.Popen(cmd, shell=True, stdout=w, 
                        stderr=subprocess.STDOUT, bufsize=0)

with open(w.name, 'r') as r:
    for line in r:
        print line
w.close()

Or any other best solutions for non blocking,unbuffering live output on windows app.

Any help would be appreciated.

Note:1.The python file which I want to run has more print statements(ie more content)

2.Windows server 2012,python 2.7

score 6 · Accepted Answer · edited May 23 '17 at 11:47

6

I understand you're frustration. It looks like you've almost come to the answer yourself.

I'm building on the answer from this SO post. But that answer doesn't use TemporaryFile and also I used the tail follow method from here which I have found to offer the fastest output to the terminal with very large volumes of output. This eliminates extraneous calls to print.

Side note: If you've got other asynchronous stuff to do then you can wrap up the code below the imports in a function and use the gevent package and import sleep from gevent and Popen, STDOUT from gevent.subprocess. It is what I'm doing and may help you avoid leftover slowdown (only reason I mention it).

import sys
from tempfile import TemporaryFile
from time import sleep
from subprocess import Popen, STDOUT

# the temp file will be automatically cleaned up using context manager
with TemporaryFile() as output:
    sub = Popen(cmd, stdout=output, stderr=STDOUT, shell=True)
    # sub.poll returns None until the subprocess ends,
    # it will then return the exit code, hopefully 0 ;)
    while sub.poll() is None:
        where = output.tell()
        lines = output.read()
        if not lines:
            # Adjust the sleep interval to your needs
            sleep(0.1)
            # make sure pointing to the last place we read
            output.seek(where)
        else:
            sys.__stdout__.write(lines)
            sys.__stdout__.flush()
    # A last write needed after subprocess ends
    sys.__stdout__.write(output.read())
    sys.__stdout__.flush()

edited May 23 '17 at 11:47

Community

1
1

answered Jul 16 '16 at 17:38

John Lunzer

361
2
5

I could not use `from __future__ import print_function`,since I have so many print in the current file.Is there any other possible way? – Nithya Jul 21 '16 at 07:38
Yes, use sys.stdout.write instead of print, removing the end argument. I will edit the answer when I have time. – John Lunzer Jul 22 '16 at 10:33
Okay, I changed the answer. I would try to use it without `sys.stdout.flush()` first and see if that works for you. I left it in for completeness. – John Lunzer Jul 22 '16 at 10:54
With `sys.stdout.flush()` attribute error is coming. ` AttributeError: 'TextCtrl' object has no attribute 'flush' `.And no changes in the python app's behavior :-( – Nithya Jul 22 '16 at 12:22
what if you remove the `sys.stdout.flush()` statements? – John Lunzer Jul 23 '16 at 16:54
Your error indicates that your `sys.stdout` got reassigned at some point earlier in the program, as the original value for `sys.stdout` __should__ have a `flush` method. I've made an edit to the posted code which will use original value for stdout. Please try that. – John Lunzer Jul 23 '16 at 17:05
It is printing on the windows command prompt(which runs in background for my python app),but not in my app console?....Usually to print anything in the app console i am using self.output.write().Here `self.output` is the `TextCtrl` – Nithya Jul 26 '16 at 11:10
What framework/package are you using for your app console? This is very likely the root of the problem. If this is printing successfully to the command prompt then the code I provided is valid. At this point you need to better understand the data structure which controls the output to your "app console". Your original question did not provide details of a console separate from the standard command prompt. You may make edits to your original question or accept this answer and ask a new one detailing your framework as the source of the problem. I suggest the latter. – John Lunzer Jul 26 '16 at 11:41
Ya this is working..I found the way to print in my app console with your code..Thanks. – Nithya Jul 26 '16 at 12:57
But still I am facing the "Not responding" issue while running the file. – Nithya Jul 26 '16 at 12:59
This fixed my hanging issue! Because STDOUT buffers to memory, it can fill up after a few pages and cause your subprocess to hang!! Writing to a temp file fixes the issue. – jonincanada May 18 '19 at 11:35

python-Can we use tempfile with subprocess to get non buffering live output in python app

1 Answers1