Python Subprocess.Popen from a thread

Question

I'm trying to launch an 'rsync' using subprocess module and Popen inside of a thread. After I call the rsync I need to read the output as well. I'm using the communicate method to read the output. The code runs fine when I do not use a thread. It appears that when I use a thread it hangs on the communicate call. Another thing I've noticed is that when I set shell=False I get nothing back from the communicate when running in a thread.

Answer 1

You didn't supply any code for us to look at, but here's a sample that does something similar to what you describe:

import threading
import subprocess

class MyClass(threading.Thread):
    def __init__(self):
        self.stdout = None
        self.stderr = None
        threading.Thread.__init__(self)

    def run(self):
        p = subprocess.Popen('rsync -av /etc/passwd /tmp'.split(),
                             shell=False,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)

        self.stdout, self.stderr = p.communicate()

myclass = MyClass()
myclass.start()
myclass.join()
print myclass.stdout

Answer 2

Here's a great implementation not using threads: constantly-print-subprocess-output-while-process-is-running

import subprocess

def execute(command):
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    output = ''

    # Poll process for new output until finished
    for line in iter(process.stdout.readline, ""):
        print line,
        output += line


    process.wait()
    exitCode = process.returncode

    if (exitCode == 0):
        return output
    else:
        raise Exception(command, exitCode, output)

execute(['ping', 'localhost'])