How to spawn another process and capture output in python?

Question

I'm just learning Python but have about 16 years experience with PERL and PHP.

I'm trying to get the output of ngrep and write it to a log file using Python while also tailing the log file. I've seen some examples online but some seem old and outdated and others use shell=True which is discouraged.

In perl I just use something similar to the following

#!/usr/bin/perl
open(NGFH,"ngrep -iW byline $filter");
while ($line = <NGFH>) {
    open(LOG,">> /path/to/file.log")
    // highlighting, filtering, other sub routine calls
    print LOG $line
}

I've gotten tail to work but ngrep doesn't. I'd like to be able to run this infinately and output the stream from ngrep to the log file after filtering. I couldn't get the output from ngrep to show in stdout so that's as far as I've gotten. I was expecting to be able to see the data file tail as the log file was updated and see the output from ngrep. For now i was just using bash to run the following.

echo "." >> /path/to/ngrep.log

Thanks!

Here's what I got so far...

Updated This seems to work now. I wouldn't know how to improve on it though.

import subprocess
import select
import re

log = open('/path/to/ngrep.log','a+',0)
print log.name

n = subprocess.Popen(['ngrep', '-iW', 'byline'],\
    stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p = select.poll()
p.register(n.stdout)

f = subprocess.Popen(['tail','-F','-n','0','/path/to/tailme.log'],\
    stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p2 = select.poll()
p2.register(f.stdout)

def srtrepl(match):
    if match.group(0) == 'x.x.x.x':
        # do something
    if match.group(0) == 'x.x.y.y':
        # do something else

    return '\033[92m'+ match.group(0) + '\033[0m'

while True:
    if p.poll(1):
        line = n.stdout.readline()
        s = re.compile(r'(8.8.(4.4|8.8)|192.168.[0-9]{1,3}.[0-9]{1,3})' )
        print s.sub( srtrepl, line )
        log.write(n.stdout.readline())

    if p2.poll(1):
        print f.stdout.readline().rstrip('\n')

Why would you need tail and ngrep? You could use "pure" Python code in here. — Kijewski, Jan 03 '16 at 01:00
@Kay Convenience, familiarity with the tools, and use of well-tested & comprehensive tools. Normally, you'd connect these tools with a (ba/z/c)sh script. Perl can do this very well, and provide easier means of handling data in between those tools. Python is not optimal for acting like a shell script. — , Jan 03 '16 at 01:04
Depending on your goals (e.g., using Python as a shell script), you might find [sh.py](http://amoffat.github.io/sh/index.html) useful, or [plumbum](https://plumbum.readthedocs.org/en/latest/). Both are externals modules, but easy to install. — , Jan 03 '16 at 01:07
Do you want to interleave log and grep output or is it okay to print one then the other? — tdelaney, Jan 03 '16 at 01:10
@Evert, Python works exceedingly well as a shell script replacement if you use it right. And for `tail -f` and `grep` using Python directly is far easier than building a subprocess pipe. — Kijewski, Jan 03 '16 at 01:12
@Kay but `ngrep` is a different beast - _"ngrep is a pcap-aware tool that will allow you to specify extended regular expressions to match against data payloads of packets"_ — tdelaney, Jan 03 '16 at 01:25
@Kay I'm just learning Python but would like to see if I convert my existing PERL script to Python and gain performance and use Python for future projects. I currently have scripts written in PERL that I use for firewall updating, allowing dymanic hosts through firewalls, irc, and other things such as locating outdated wordpress and joomla installs and php file fingerprinting. — Steve Hernandez, Jan 03 '16 at 01:26
The PERL code I posted is a portion of code that is used for detecting SIP attacks against Asterisk PBX servers. It uses ngrep to watch traffic on :5060 then analyses the packet and decides weather or not to add firewall rules to the system. It also contacts a central server that holds offending ip addresses reported from other systems so PBX servers can update their firewalls. The central server has an IRC bot I wrote that tails the log file of the reports and publishes them on a private IRC server. — Steve Hernandez, Jan 03 '16 at 01:31
Do you just want the output to go to `stdout`? If so, skip `stdout=subprocess.PIPE` in your calls, wait for `ngrep` to complete and then kill `tail`. — tdelaney, Jan 03 '16 at 01:36
tdelaney, I didn't read the man page that far. I thought ngrep was an unholy combination of curl and grep. :) OP, what makes you think that Python was more performant than Perl? Perl is a fine language if you use it correctly. I don't use it because it is easier to shoot yourself in the foot and harder on the eyes (IMO), but maybe I am just lazy. Who knows? Converting your example is easy enough, but first I'd like to know if it would be really helpful to you or (at least) other readers of the question. — Kijewski, Jan 03 '16 at 01:39

score 1 · Answer 1 · edited May 23 '17 at 11:52

To emulate your perl code in Python:

#!/usr/bin/env python3
from subprocess import Popen, PIPE

with Popen("ngrep -iW byline".split() + [filter_], stdout=PIPE) as process, \
     open('/path/to/file.log', 'ab') as log_file:
    for line in process.stdout: # read b'\n'-separated lines
        # highlighting, filtering, other function calls
        log_file.write(line)

It starts ngrep process passing filter_ variable and appends the output to the log file while allowing you to modify it in Python. See Python: read streaming input from subprocess.communicate() (there could be buffering issues: check whether ngrep supports --line-buffered option like grep and if you want to tail file.log then pass buffering=1 to open(), to enable line-buffering (only usable in the text-mode) or call log_file.flush() after log_file.write(line)).

You could emulate ngrep in pure Python too.

If you want to read output from several processes concurrently (ngrep, tail in your case) then you need to able to read pipes without blocking e.g., using threads, async.io.

How to spawn another process and capture output in python?

1 Answers1

Linked