0

I have a python script search for logs, it continuously output the logs found and I want to use linux pipe to filter the desired output. example like that:

$python logsearch.py | grep timeout

The problem is the sort and wc are blocked until the logsearch.py finishes, while the logsearch.py will continuous output the result.

sample logsearch.py:

p = subprocess.Popen("ping google.com", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for line in p.stdout:
    print(line)

UPDATE:

figured out, just change the stdout in subprocess to sys.stdout, python will handle the pipe for you.

p = subprocess.Popen("ping -c 5 google.com", shell=True, stdout=**sys.stdout**)

Thanks for all of you help!

Vincent Mao
  • 201
  • 2
  • 8
  • `ping -c 5` is an odd example of a script that runs forever! But, yeah, you can't sort things or count the number of things until you have them all. – tdelaney Oct 09 '16 at 21:01
  • @tdelaney I removed the count for ping and instead of sort, I use grep so suppose the data should flow to the grep through pipe but it seems runs forever without print anything. – Vincent Mao Oct 09 '16 at 21:22
  • You don't receive nothing because the output don't match with timeout. Try $python logsearch.py | grep -v timeout – Jose Raul Barreras Oct 09 '16 at 21:32

1 Answers1

1

And why use grep? Why don't do all the stuff in Python?

from subprocess import Popen, PIPE
p = Popen(['ping', 'google.com'], shell=False, stdin=PIPE, stdout=PIPE)

for line in p.stdout:
    if 'timeout' in line.split():
        # Process the error
        print("Timeout error!!")
    else:
        print(line)

UPDATE:
I change the Popen line as recommended @triplee. Pros and cons in Actual meaning of 'shell=True' in subprocess

Community
  • 1
  • 1
Jose Raul Barreras
  • 849
  • 1
  • 13
  • 19
  • the reason need grep or other shell cmd is to add the flexibility, some time we might want add grep or sort or awk in order to better process the data. – Vincent Mao Oct 09 '16 at 21:47
  • OK. Just in case, you can use the [Python module sh](https://pypi.python.org/pypi/sh). Implements a lot bash useful commands. – Jose Raul Barreras Oct 10 '16 at 08:32
  • Also use `Popen(['ping', 'google.com'], shell=False, ...)` to avoid spawning a useless shell. – tripleee Oct 10 '16 at 19:22