Capture / redirect all output of ProcessPoolExecutor

Question

I am trying to capture all output from a ProcessPoolExecutor.

Imagine you have a file func.py:

print("imported")  # I do not want this print in subprocesses

def f(x):
    return x

then you run that function with a ProcessPoolExecutor like


from concurrent.futures import ProcessPoolExecutor
from func import f  # ⚠️ the import will print! ⚠️

if __name__ == "__main__":
    with ProcessPoolExecutor() as ex:  # ⚠️ the import will happen here again and print! ⚠️
        futs = [ex.submit(f, i) for i in range(15)]
        for fut in futs:
            fut.result()

Now I can capture the output of the first import using e.g., contextlib.redirect_stdout, however, I want to capture all output from the subprocesses too and redirect them to the stdout of the main process.

In my real use case, I get warnings that I want to capture, but a simple print reproduces the problem.

This is relevant to prevent the following bug https://github.com/Textualize/rich/issues/2371.

Capture / redirect all output of ProcessPoolExecutor

0 Answers0