Subprocess.Popen get output even in case of timeout

Question

I am using subprocess.Popenand Popen.communicate to run a process with a timeout, similar to the suggestion given is this question:

Subprocess timeout failure

Thus I am using the following code:

with Popen(["strace","/usr/bin/wireshark"], stdout=subprocess.PIPE,stderr=subprocess.PIPE, preexec_fn=os.setsid) as process:
    try:
        output  = process.communicate(timeout=2)[0]
        print("Output try", output)
    except TimeoutExpired:
        os.killpg(process.pid, signal.SIGINT)  # send signal to the process group
        output = process.communicate()[0]
        print("Output except",output)
return output

And I get the following output:

Output except b''

and b'' as a return value.

How can I get the output of the process (the output until it is killed) even though the TimeoutExpired exception is raised?

After you have killed the process with os.killpg, the process and all information of it is gone. You cannot communicate anymore to it. If you want to communicate to it, you must use process.kill(). Then the process object is killed it in a way that the output is still available. This is also what the documentation of subprocess recommends. So the other two answers here are good. The alternative implementation that Booboo placed is not required. It is unclear why he proposes this. — habrewning, Oct 20 '22 at 19:50

Booboo · Answer 1 · 2022-10-19T12:43:52.053

Use kill to terminate the process:

import subprocess
import os

def foo():
    with subprocess.Popen(["strace", "/usr/bin/wireshark"], stdout=subprocess.PIPE,stderr=subprocess.PIPE, preexec_fn=os.setsid) as process:
        try:
            return process.communicate(timeout=2)[0]
        except subprocess.TimeoutExpired:
            process.kill()
            return process.communicate()[0]

output = foo()

If this does not work for you (it does for me), then:

Instead of calling communicate, start a thread that will loop reading chunks from process.stdout and write them to a passed queue.Queue instance. The main thread will now call process.wait(timeout=2) to await the termination. If subprocess.TimeoutExpired is raised, then the process will be killed. When we know that the process is no longer running and the thread has written all the chunks it will ever write, we loop doing calls to get_nowait on the queue to get all the chunks until the queue is empty and finally we concatenate the chunks together:

import subprocess
from threading import Thread
from queue import Queue
import os

def foo():

    def reader(stream, queue):
        while True:
            data = stream.read(4096)
            if data == b'':
                break
            queue.put(data)

    with subprocess.Popen(["strace", "/usr/bin/wireshark"], stdout=subprocess.PIPE,stderr=subprocess.PIPE, preexec_fn=os.setsid) as process:
        queue = Queue()
        t = Thread(target=reader, args=(process.stdout, queue))
        t.start()
        try:
            process.wait(timeout=2)
        except subprocess.TimeoutExpired:
            process.kill()
        # Never do a join on a process writing to a multiprocessing.Queue
        # before you retrieve all items from the queue. But here we are OK:
        t.join()
        output = []
        try:
            while True:
                output.append(queue.get_nowait())
        except Empty:
            pass
        return b''.join(output)

output = foo()

score 0 · Answer 2 · answered Jun 02 '17 at 18:56

Popen objects have a kill method, why not use that instead? After a process is killed, you can just read it's stdout. Does this work?

with Popen(["strace","/usr/bin/wireshark"], stdout=subprocess.PIPE,stderr=subprocess.PIPE, preexec_fn=os.setsid) as process:
    try:
        output  = process.communicate(timeout=2)[0]
        print("Output try", output)
    except TimeoutExpired:
        process.kill()
        output = process.stdout.read().decode()
        print("Output except",output)
return output

Unfortunately not, since this does not kill the process group - thus the python script is blocked. If I kill the process manually, the output variable remains empty. — Sleik, Jun 03 '17 at 10:54

score -1 · Answer 3 · answered Oct 14 '22 at 07:18

According to the document: Popen.communicate

If the process does not terminate after timeout seconds, a TimeoutExpired exception will be raised. Catching this exception and retrying communication will not lose any output.

Your code is worked perfectly for me, the problem is that your command is write log to stderr only so the output is empty. See this Determine if output is stdout or stderr

Subprocess.Popen get output even in case of timeout

3 Answers3