4

I have a simple python program:

test.py:

import time
for i in range(100000):
    print i
    time.sleep(0.5)

I want to use another program that executes the above one in order to read the last line output while the above program is counting.

import subprocess

process = subprocess.Popen("test",stdout=PIPE)
sleep(20) # sleeps an arbitrary time
print stdout.readlines()[-1]

The problem is that process.stdout.readlines() waits until test.py finishes execution. Is there any way to read the last line that has been writen in the output while the program is executing?

Alejandro Garcia
  • 1,171
  • 1
  • 11
  • 22
  • If your "test" program is also written in python, you should consider using threads and pass information from one side to the other instead of printing and reading from a file... – JBernardo Nov 11 '12 at 14:31
  • @JBernardo Thanks for the suggestion, but im what im trying to archieve is to make it work for any program. – Alejandro Garcia Nov 11 '12 at 14:46

2 Answers2

3

You could use collections.deque to save only the last specified number of lines:

#!/usr/bin/env python
import collections
import subprocess
import time
import threading

def read_output(process, append):
    for line in iter(process.stdout.readline, ""):
        append(line)

def main():
    process = subprocess.Popen(["program"], stdout=subprocess.PIPE)
    # save last `number_of_lines` lines of the process output
    number_of_lines = 1
    q = collections.deque(maxlen=number_of_lines)
    t = threading.Thread(target=read_output, args=(process, q.append))
    t.daemon = True
    t.start()
    #
    time.sleep(20)

    # print saved lines
    print ''.join(q),
    # process is still running
    # uncomment if you don't want to wait for the process to complete
    ##process.terminate() # if it doesn't terminate; use process.kill()
    process.wait()

if __name__=="__main__":
    main()

See other tail-like solutions that print only the portion of the output

See here if your child program uses a block-buffering (instead of line-bufferring) for its stdout while running non-interactively.

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • Nice answer, but sadly im having the same problem that I stated before. It only displays output if the called process has finished – Alejandro Garcia Nov 11 '12 at 14:44
  • I believe in `test.py` Python is buffering the output sent to `sys.stdout`. I got this solution to work by using the `-u` flag: `python -u test.py`. (Otherwise the output is empty.) – unutbu Nov 11 '12 at 14:45
  • @unutbu: yes. I've forgot to mention it. I've added the link that provides some workarounds – jfs Nov 11 '12 at 14:52
  • @AlejandroGarcia: You can avoid the IOError by putting `process.terminate()` at the end of `main`. – unutbu Nov 11 '12 at 14:54
1

Fairly trivial with sh.py:

import sh

def process_line(line):
    print line

process = sh.python("test.py", _out=process_line)
process.wait()
amoffat
  • 696
  • 4
  • 12