0

I want to redirect the output of (potentially multiple) processes started from within a python script to both stdout and a log file, pretty much like as the unix tool tee does it. A the processes are rather long-lasting I would like to show/log the line when it is printed. Line buffering is ok but waiting for the process to finish before anything is printed to stdout is not an option.

I cannot find a way to achieve that though. When looking at subprocess.Popen() I see that I can forward the output from stdout to a pipe, to a file descriptor or to a file object. It seems that I have to wait until the process ends when calling communicate() to read from the pipe. I neither have a file descriptor and faking a file object does not work either as Popen seems to need a real file.

The only solution that I know so far is reading line-by-line from Popen as suggested in Python Popen: Write to stdout AND log file simultaneously. However, this works for a single process only. I would like to handle multiple processes launched from the python script and all should print to stdout and into the same log.

I especially focused on using subprocess.Popen. Qt's QProcess seems to be a potential solution by letting callbacks do the job but unfortunately I do not have a GUI application and I do not know how to run a Qt event loop within a linearly executed python script.

Is there any other way to capture output to stdout in a log file and at same time having live output to stdout?

Thanks!

Community
  • 1
  • 1
Stefan
  • 1,131
  • 2
  • 12
  • 30
  • *"does not work either"* is not informative. What have you tried to run multiple processes concurrently (`threading`, `asyncio`)? What did you expect to happen? What happened instead? describe step by step. – jfs Apr 30 '16 at 16:24
  • This is not a bug report. I am asking for conceptual help. If you read the suggested solution in the question I cite (http://stackoverflow.com/a/15535389/1557304) you see that this holds for a single process only. I have not found a solution to receive the output of multiple processes. – Stefan May 01 '16 at 15:00
  • I updated my question to describe what I've tried so far. – Stefan May 01 '16 at 15:05
  • Why do you expect an answer to the question that doesn't mention multiple processes, to handle multiple processes? Are you looking for a simple answer e.g., "use threads" or "use asyncio"? Do you need help on how to start a thread in Python? – jfs May 01 '16 at 15:08
  • Hm, actually it was mentioned but may be not apparent enough. I updated the question to stress this fact. Your comment also points me to a solution where actually separate threads might actively poll a process and forward the output to another function. Thanks! I will look into that. – Stefan May 01 '16 at 16:08
  • 1
    If you need a code example then [follow links mentioned in this question](http://stackoverflow.com/q/25750468/4279) e.g., it is easy to modify `teed_call()` function to start more than one process and even easier to start multiple processes using [`asyncio` version](http://stackoverflow.com/a/25960956/4279) – jfs May 01 '16 at 17:13
  • 1
    I updated my answer to address your comment. – Vinay Sajip May 01 '16 at 18:59
  • @Vinay Sajip: Thank you! – Stefan May 01 '16 at 22:55

1 Answers1

1

You could use a third-party library like sarge (disclosure: I'm the maintainer) which allows you to open multiple subprocesses, capture their output streams, read from them a line at a time and do what you want with those lines. You need to be aware of things like process buffering issues - have a look at this section of the docs for more information.

Update: In response to your comment - yes, using sarge you can start multiple processes (using async=True), and then you have to actively poll their output streams in the parent process.

Vinay Sajip
  • 95,872
  • 14
  • 179
  • 191
  • Does that mean if start two processes in Python with sarge I can received the output of both within my script? Does that also mean I have to actively poll both processes by repeatedly calling `readline()`? – Stefan May 01 '16 at 15:02