Processing realtime output of access_log with subprocess

Question

I am trying to accomplish an idea which i really don't know how to do it. Basically i am trying to capture the value through grep command as shown below

p = subprocess.Popen('tail -f /data/qantasflight/run/tomcat/logs/localhost_access_log.2016-02-29.txt  | grep /qantas-ui/int/price?', stdout=subprocess.PIPE, shell = True)
stdout = p.communicate()[0]

Process the stdout value and then push the value as shown below

f = urllib.urlopen("http://162.16.1.90:9140/TxnService", params2)

param2 is the value where i will process the result given by the subprocess.Popen

In a nutshell i want the following :

-- Wait for the new value --> -- Process the value --> -- Push the value -->

This should be realtime and the python script will keep on getting the new value, process it and then push the value.

1- why do you use external processes such as `tail`, `grep` here (especially `grep`)? 2- related: [Python: read streaming input from subprocess.communicate()](http://stackoverflow.com/q/2715847/4279) — jfs, Mar 02 '16 at 19:16
To clarify, you could use something like `if '/qantas-ui/int/price' in line:` instead of `grep` here. `tail -f` could be replaced with [`watchdog` package (`$ pip install watchdog`)](http://pythonhosted.org/watchdog/quickstart.html) though it might be simpler to call `tail -f`. — jfs, Mar 02 '16 at 19:27

score 1 · Accepted Answer · answered Mar 02 '16 at 13:47

1

i try code somethink as you want, but im only printing new lines in log, you can process line and push

from __future__ import print_function
import subprocess
from time import sleep

f = open('test.log', 'r')
while True:
    line = ''
    while len(line) == 0 or line[-1] != '\n':
        tail = f.readline()
        if tail == '':
            continue
        line = tail

    print(line, end='')

this print new line to console, just edit and use :) maybe i help you :)

answered Mar 02 '16 at 13:47

sunny

178
3
12

your code makes sense but when i run it then it opens the entire file and then starts reading the latest end lines. What modification should i do if i want to read those last latest lines which contains "/qantas-ui/int/price?" – fear_matrix Mar 03 '16 at 07:07
if you want read only new lines, just after `f = open(...)` write on new line this `f.seek(0, 2)` and then, when in log is appended new line, just parse it and process as you need – sunny Mar 03 '16 at 07:25
let me try and i will get back to you. – fear_matrix Mar 03 '16 at 08:41

Processing realtime output of access_log with subprocess

1 Answers1