1

What is the proper way of reading subprocess and the stdout

Here are my files:

traffic.sh
code.py

traffic.sh:

sudo tcpdump -i lo -A | grep Host:

code.py:

proc = subprocess.Popen(['./traffic.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
# Do some network stuff like ping places, send an email, open a few web pages and wait for them to finish loading
# Stop all traffic and make sure its over
data = proc.stdout.read()
proc.kill()

The code above sometimes works and sometimes doesnt.

The times that it fails, its is due to getting stuck on the proc.stdout.read().

I have followed a bunch of examples that recommend to setup a thread and queue for the proc and read the queue as the proc writes. However, this turnout to be intermittent as to how it works.

I feel like im doing something wrong with the kill and the read. because I can guarantee that there is no communication happening on lo when I make that call and therefore, traffic.sh should not be printing out anything at all.

Then why is the read blocking.

Any clean alternative to the thread?

Edit I have also tried this, in the hope that the read would no longer block since the process would is terminated

proc = subprocess.Popen(['./traffic.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
# Do some network stuff like ping places, send an email, open a few web pages and wait for them to finish loading
# Stop all traffic and make sure its over
proc.kill()
data = proc.stdout.read()
Cripto
  • 3,581
  • 7
  • 41
  • 65
  • The read continues until the input stream is closed. Nothing obviously there closes the output stream from the script (thus, the input stream to the subprocess) while the pipeline in the script is still running. – Charles Duffy Jul 18 '13 at 01:38
  • What's your actual desired behavior? Maybe you want to only read one line, instead of reading all input? If that's the case, have your pipeline end in `head -n 1`. – Charles Duffy Jul 18 '13 at 01:39
  • @CharlesDuffy, can I force the pipe to close, I also did `proc.kill()` before the read hoping it would close. but had the same issue – Cripto Jul 18 '13 at 01:41
  • Ahh -- so there's a delay first. Yes, you need to do the `kill()` before you can do a blocking read. Though the blocking read is a bloody bad idea even so -- it means that tcpdump's output is limited by your buffer size. – Charles Duffy Jul 18 '13 at 01:47
  • @CharlesDuffy. No go. :( Note my edit please. – Cripto Jul 18 '13 at 01:53
  • Your edit doesn't make sense. What does ongoing activity have to do with signal handling? – Charles Duffy Jul 18 '13 at 01:54
  • The read is blocking because `.read()` is _always_ blocking, by definition, unless you've changed the source it's reading from to be non-blocking. By default, blocking is exactly what it's supposed to do. If you didn't want it to block, you'd need to be in async mode. – Charles Duffy Jul 18 '13 at 01:55
  • @CharlesDuffy Sorry. I guess Im not understudying the purpose of the kill. My intent is to get all of the data traffic.sh has produced (printed to stdout) to be save in a veritable. When I make this call, I dont care what comes in the feature. I want what has already been printed out. – Cripto Jul 18 '13 at 01:57
  • Follow the answer at http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python – Charles Duffy Jul 18 '13 at 01:58
  • The read doesn't block until things that are actively being printed finish printing -- it blocks until the stream it's reading from is closed. So, as long as `grep` is still running, `read()` won't return, even if there's nothing being actively printed, unless you go into async mode. Which the question I linked you to is directly on-point for. Make sense, now? – Charles Duffy Jul 18 '13 at 02:00
  • I had followed that link before I posted my question with the same results. Thats why I noted "Any clean alternative to the thread?" – Cripto Jul 18 '13 at 11:06
  • Yes, the alternative is async I/O. Which some of the answers other than the accepted one in that link discuss. Not everything there uses threads. – Charles Duffy Jul 18 '13 at 11:53
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/33692/discussion-between-charles-duffy-and-user1048138) – Charles Duffy Jul 18 '13 at 11:56

0 Answers0