2

I think I might be repeating the question but I didn't find any of the answers suited to my requirement. Pardon my ignorance.

I have a program running which continuously spits out some binary data from a server.It never stops until it's killed.

I want to wrap it in a python script to read the output and process it as and when it arrives. I tried out few of the subprocess ideas in stack overflow but no use. Please suggest.

p=subprocess.popen(args,stderr=PIPE,stdin=PIPE,stdout=PIPE,shell=FALSE)
#p.communicate#blocks forever as expected
#p.stdout.read/readlines/readline-->blocks
#select(on p.stdout.fileno())-->blocks

what is the best method?

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
rroh
  • 149
  • 1
  • 2
  • 10
  • I would like to add that the data generated is a http stream.Forgot to mention that – rroh Nov 14 '11 at 05:05
  • OK.matter finally resolved.I figured out that the server feeding the data was intermittently failing to stream as in no HTTP error but just hanging in wait state.It was slow sometimes and sometimes failing to authenticate.The solution read/readline by icktoofay is working fine. – rroh Nov 14 '11 at 06:49
  • I was able to solve another related problem using asyncproc module mentioned in this post http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python.Even though read no more blocks by using select,I need to read the exact bytes the app spits out, otherwise I would have to do a packet re-construction logic which the app already did.To circumvent, that using asyncproc helped.Now doing a read returns the exact bytes(one packet length) the other app spat at a time. – rroh Nov 15 '11 at 23:14

3 Answers3

3

Read with a length limit:

proc = subprocess.Popen(args, stdin=None, stdout=subprocess.PIPE, stderr=None)
while True:
    chunk = proc.stdout.read(1024)
    # chunk is <= 1024 bytes

This is the code from your comment, slightly modified. It works for me:

import subprocess

class container(object):
    pass

self = container()
args = ['yes', 'test ' * 10]

self.p = subprocess.Popen(args, stdin=None, stderr=None,
                          stdout=subprocess.PIPE, shell=False)
while True:
    chunk = self.p.stdout.read(1024)
    print 'printing chunk'
    print chunk
icktoofay
  • 126,289
  • 21
  • 250
  • 231
  • this didn't work: {self.p=subprocess.Popen(args,stdin=None,stderr=None,stdout=subprocess.PIPE,shell=False) #out,err=p.communicate() print self.p print self.p.stdout while True: chunk=self.p.stdout.read(1024) self.p.stdout.flush() print 'printing chunk' print chunk} – rroh Nov 14 '11 at 04:40
  • This code looks correct. @rroh can you be more precise about how "it didn't work"? – Raymond Hettinger Nov 14 '11 at 04:46
  • 1
    Mine just waits indefinitely in read() never returns. ', mode 'rb' at 0x8f0ad30> – rroh Nov 14 '11 at 04:59
  • I can see the the running process(from popen) using ps.and it does catch my keyboardinterrupt and exits.But doesn't spit out anything while the same command works fine in command line. – rroh Nov 14 '11 at 05:03
0

You could run the other program with its output directed to a file and then use Python's f.readline() to tail the file.

Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
  • Hi Raymond,I thought of that.But the program spitting data spits it as and when a packet arrives and I need to capture the entire packet and process/redirect.It would be cumbersome to write a logic for packet reconstruction from readline again.I just have the logic to process the packet and not to assemble. – rroh Nov 14 '11 at 04:56
  • I was thinking of doing trying one more method../datagenerator | myprogram.py.Any thoughts.I will try this out though. – rroh Nov 14 '11 at 05:01
0

It sounds like you could use an asynchronous version of the subprocess module. For more information, check out the developer's blog.

srgerg
  • 18,719
  • 4
  • 57
  • 39