Run and get output from a background process

Question

I already know there is several questions targeting this theme, but none of them solved my specific problem. Or at least I could not find it.

I need to execute some program in background, wait for output and manipulate it. But the background program must keep executing.

The info from the background program that I need is located, precisely, in the second line of its output. There is no problem if this program blocks my code until reach this line. But it is essential that it unlocks after that line so I can perform other tasks completely unrelated with the background program.

Still, I cannot figure out how to accomplish this. I've read a lot the documentation of subprocess module, in particular, subprocess.Popen.

Being pratical: why this code does not work with ['localtunnel', '8000'] argument? It outputs nothing...

I know I do not need root privileges to execute this.

EDIT after answers from jadkik94 and fest

Unfortunately, both answers do not work for me. Maybe I am doing something wrong...

First of all. A 'sanity check':

import subprocess, threading, time
can_break = False

def run():
    args = ["ping", "google.com"]
    popen = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE)
    while not can_break:
        print popen.stdout.readline()

t = threading.Thread(target=run)

try:
    t.start()
    while True:
        print 'Main thread...'
        time.sleep(1)
except KeyboardInterrupt:
    can_break = True

The code above works properly with output similar to this:

Main thread...
PING google.com (74.125.234.160) 56(84) bytes of data.
64 bytes from plusone.google.com (74.125.234.160): icmp_req=1 ttl=54 time=82.5 ms
Main thread...
64 bytes from plusone.google.com (74.125.234.160): icmp_req=2 ttl=54 time=82.7 ms
[...]

But when I use it with the args I want (args = ['localtunnel', 8000]), the only output is Main thread....

When I call localtunnel in the main thread (blocking), it returns the desired output:

In [x]: popen = subprocess.Popen(['localtunnel', '8000'])
  This localtunnel service is brought to you by Twilio.
  Port 8000 is now publicly accessible from http://????.localtunnel.com ...

This approach is based on jadkik94's answer. But fest's answer does not work either.

To be clear: Does the Python script need any of the background program's output beyond the second line? Or does it just need to let it keep running? — David Robinson, May 15 '12 at 16:27
After the second line, the background program can "hide" its output. — borges, May 15 '12 at 16:32

score 1 · Answer 1 · edited May 23 '17 at 10:33

To launch a program in a non blocking way but still being able to see the output of program, the program has to be launched in a separate thread or process. Ryan has posted a nice code sample here: Python Subprocess.Popen from a thread

Keep in mind that the last line print myclass.stdout will print the output how it appears at that time. If program is just being launched, it might not have output anything at all, so your code should probably read from myclass.stdout until it receives the line you need.

score 0 · Answer 2 · answered May 15 '12 at 17:14

0

You can run it in a thread (so that it doesn't block your code from running), and get the output until you get the second line, then wait for it to terminate. This is an example that will read the output from the command dir /s on Windows to get all the directory listing.

import subprocess, thread, time

def run():
    global can_break

    args = ["dir", "/s"]
    shell = True

    count = 0
    popen = subprocess.Popen(args, shell=shell, stdout=subprocess.PIPE)
    while True:
        line = popen.stdout.readline()
        if line == "": continue
        count += 1
        if count == 2:
            do_something_with(line)
            break

    print "We got our line, we are waiting now"
    popen.wait()
    print "Done."
    can_break = True

def do_something_with(line):
    print '>>> This is it:', line

thread.start_new_thread(run, tuple())

can_break = False
while not can_break:
    print 'Wait'
    time.sleep(1)

print 'Okay!'

Output will look like:

Wait
>>> This is it:  Volume Serial Number is XXXX-XXXX

We got our line, we are waiting now
Wait
Wait
Wait
.
.
.
Done.
Wait
Okay!

answered May 15 '12 at 17:14

jadkik94

7,000
2
30
39

It seems you have a problem with either 1) the fact that it is run in a thread (we have a problem) 2) the parameters you passed to `subprocess.Popen`. So try running it in a thread, just like you did in the main one: `subprocess.Popen(args)` – jadkik94 May 17 '12 at 08:32
One more thing: The output with args is "Main Thread..." with new lines or without? There should be a ton of new lines, because `popen.stdout.readline()` should return something... and you're printing it. I'll try to get `localtunnel` running here, and see by myself. – jadkik94 May 17 '12 at 09:23
There is no new lines. I think this is because the `popen.stdout.readline()` blocks until it has data. But a strange thing I just noticed now is that when I kill the program after several `Main thread...`, it appears the first line of localtunnel: `This service is localtunnel Brought to you by Twilio.`. But just when I kill the whole program (with CTRL+C). – borges May 17 '12 at 13:01
When I put just `subprocess.Popen(args)` in the second thread it works ([code here](http://pastebin.com/K8yYQUi1)). But now I do not know how to capture and read the output. – borges May 17 '12 at 13:06
I forgot to add in my first comment: That seems to suggest that popen output is, somehow, buffered. – borges May 17 '12 at 13:12
Try this: http://pastebin.com/tdCwMnHF. Sorry there's an error there. Use `popen.stdout.readline()` – jadkik94 May 17 '12 at 13:13
Sorry, did not work. I will upvote your reply (maybe it works for other people with similar problems) and thank you for your great effort in helping me! – borges May 17 '12 at 13:40

Run and get output from a background process

2 Answers2