Stream input from terminal to Python script

Question

I have a log file on my server and I am using a CLI Program to fetch the content to terminal. I need to do a bit of filtering and json operation and I am more comfortable in doing that in python rather than in some bash script. Now my question is is there a way to pipe the stream to python?

something like this

cliProgram fetchLogs | python script.py

In Python, I want to parse the content line by line so python file should have a way to read the data line by line and if data is not available (may be because of network delay) , it should wait for more data and exit only when the stream is closed.

I have already gone through the answers in [other question](http://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin-in-python) . The question was more towards the streaming the content. If I am retrieving the content over wire, where the content comes as a stream in chunks, will the python script wait until the content is available? — Vivek Kumar, May 11 '17 at 10:20
This isn't a Python thing, it's a shell thing. This is how pipelines always work, for all programs reading from stdin/writing to stdout. Having said that, the shell is a platform feature, so you should probably mention which platform you're using. — Useless, May 11 '17 at 10:27

score 4 · Answer 1 · answered May 11 '17 at 10:13

You just have to iterate on sys.stdin :

bruno@bigb:~/Work/playground$ cat pipein.py
import sys

def main():
    for line in sys.stdin:
        print "line '%s'" % line.rstrip("\n")

if __name__ == "__main__":
    main()

bruno@bigb:~/Work/playground$ cat wotdata.txt 
E = 0
m = 1
J = 3
K = 2
p = {0: 0.696969696969697, 1: 0.30303030303030304}
UDC = {(0, 1): 9.0, (1, 2): 10.0, (0, 0): 5.0, (1, 1): 6.0, (1, 0): 9.0, (0, 2): 6.0}
UDU = {(0, 1): 5.0, (1, 2): 4.0, (0, 0): 2.0, (1, 1): 4.0, (1, 0): 1.0, (0, 2): 3.0}
UAC = {(0, 1): 1.0, (1, 2): 0.0, (0, 0): 2.0, (1, 1): 3.0, (1, 0): 4.0, (0, 2): 0.0}
UAU = {(0, 1): 9.0, (1, 2): 10.0, (0, 0): 5.0, (1, 1): 6.0, (1, 0): 10.0, (0, 2): 6.0}

bruno@bigb:~/Work/playground$ cat wotdata.txt | python pipein.py
line 'E = 0'
line 'm = 1'
line 'J = 3'
line 'K = 2'
line 'p = {0: 0.696969696969697, 1: 0.30303030303030304}'
line 'UDC = {(0, 1): 9.0, (1, 2): 10.0, (0, 0): 5.0, (1, 1): 6.0, (1, 0): 9.0, (0, 2): 6.0}'
line 'UDU = {(0, 1): 5.0, (1, 2): 4.0, (0, 0): 2.0, (1, 1): 4.0, (1, 0): 1.0, (0, 2): 3.0}'
line 'UAC = {(0, 1): 1.0, (1, 2): 0.0, (0, 0): 2.0, (1, 1): 3.0, (1, 0): 4.0, (0, 2): 0.0}'
line 'UAU = {(0, 1): 9.0, (1, 2): 10.0, (0, 0): 5.0, (1, 1): 6.0, (1, 0): 10.0, (0, 2): 6.0}'

Stream input from terminal to Python script

1 Answers1