Reading from pipe in python is imposiible

Question

Hello I have the following code in python 2.6:

command = "tcpflow -c -i any port 5559"
port_sniffer = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1, shell=True)
while True:
    line = port_sniffer.stdout.readline()
    #do some stuff with line

The purpose of this code is to sniff the traffic between two processes (A and B) that communicate on port 5559.

Now let me describe the different scenarios I am having:

1) Code above is not running: A and B are communicating and i can see it clearly using logs and the linux command netstat -napl | grep 5559 shows that the processes are communicating on the desired port.

2) Code above is not running and I am sniffing by running tcpflow -c -i any port 5559 directly from shell: I can see the communication on console clearly :-).

3) Code above is running: Proccesses can't communicate. netstat -napl | grep 5559 prints nothing and logs give out errors!!!

4) Code above is running in debug mode: I can't seem to be able to step after the line line = port_sniffer.stdout.readline()

I tried using an iterator instead of a while loop (not that it should matter but still I am pointing it out). I also tried different values for bufsize (none, 1, and 8).

Please help!!

tcpflow is not a standard program. Can you give background as to what `tcpflow -c -i any port 5559` does? Does it only dump data to a file, or does it output information to stdout/stderr? — FuriousGeorge, Jan 20 '14 at 18:41
Oh, and FWIW http://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess — FuriousGeorge, Jan 20 '14 at 18:46
tcpflow captures tcp packets and writes their message body (without tcp overhead) to files or console. The -c flag makes it write its output to console. I'd rather use shell=True because I don't know what other environment can run this program. — whomaniac, Jan 21 '14 at 09:04
The problem was actually not in this code. The code I wrote works — whomaniac, Jan 26 '14 at 13:09

wnnmaw · Answer 1 · 2014-01-20T18:47:59.780

So after a quick read through the docs I found these two sentences:

On Unix, if args is a string, the string is interpreted as the name or path of the program to execute

and

The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.

Based on this, I would recommend recreating your command as a list:

command = ["tcpflow -c", "-i any port 5559"]   #I don't know linux, so double check this line!!

The general idea is this (also from the docs):

If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

Additionally, it seems that to read from your process, you should use communicate(). So

while True:
    line = port_sniffer.stdout.readline()

would become

while True:
    line = port_sniffer.communicate()[0]

But keep in mind this note from the docs:

Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

Thanks for your input but I don't think this solves the issue. Using a list or a string does the same thing. As for communicate, I haven't tried it because I know it is blocking. What I want to do is to read from stdout of tcpflow while it is running. I don't want for it to finish! — whomaniac, Jan 21 '14 at 09:08

score 1 · Answer 2 · answered Jan 21 '14 at 14:15

If I had to guess, I think the problem that you're having is that you aren't running your program as root. TCPFlow needs to be run as a privelaged user if you want to be able to sniff other people's traffic (otherwise that'd be a serious security vulnerability). I wrote the following programs and they worked just fine for your scenario

server.py

#!/usr/bin/python
import socket
s = socket.socket()
host =  socket.gethostname()
port = 12345
s.bind((host,port))
s.listen(5)
while True:
    c, addr = s.accept()
    print 'Connection from', addr
    c.send('Test string 1234')
    c.recv(1024)
    while x != 'q':
        print "Received " + x
        c.send('Blah')
        x = c.recv(1024)
    print "Closing connection"
    c.close()

client.py

#!/usr/bin/python
import socket, sys
from time import sleep
from datetime import datetime
s = socket.socket()
host = socket.gethostname()
port = 12345
s.connect((host,port))
c = sys.stdin.read(1) # Type a char to send to initate the sending loop
while True:
   s.send(str(datetime.now()))
   s.sleep(3)
   msg = s.recv(1024)

flow.py

#!/usr/bin/python
import subprocess
command = 'tcpflow -c -i any port 12345'
sniffer = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
while True:
   print sniffer.stdout.readline()

Reading from pipe in python is imposiible

2 Answers2