0

I want to capture the output of a process called with

cmd = "doit input={input} conf={conf} output_dir={dir} gpu={gpu} --db-file .doit_gpu{gpu_id}.db".format(**kwargs)
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)

continuously. This process calls functions that print their result in the console as dictionary strings. E.g.:

{'subprocess_1': [<some data>]}
# then after some time
{'subprocess_2': [<some data>]}
# and some seconds later
{'subprocess_3': [<some data>]}

I want to be able to continuously capture the output and redirect it immediately to another dictionary that I initialized before.

process_table = {}

So this is how the dictionary would have to look like after the whole process finished:

process_table = {
    {'subprocess_1': [<some data>]},
    {'subprocess_2': [<some data>]},
    {'subprocess_3': [<some data>]}
}

The key point here (which is why I don't seem to find any related questions) is that process_table should be updated immediately after a subprocess finishes, not only after the entire process finished. This is why this answer here is not enough:

for line in process.stdout:
    print("this is the output:", line,)
  • So to be clear, you want your python script to block while waiting on each subprocess to finish, then gather the stdout of the process? – lancew Aug 29 '17 at 14:49
  • Yes. But as it prints to `stdout`, I want to capture the output and redirect it to the variable `process_table`. I use two threads, one running `process` and one reading `process_table` and doing some stuff with it. –  Aug 29 '17 at 15:06

1 Answers1

0

I have found the solution to my problem. A simple

sys.stdout.flush()

in the executed program (i.e. my doit script) printed the subprocess outputs immediately and I could save them to the process_table with

for line in process.stdout:
    subprocess_output = json.loads(line.decode("utf-8"))
    process_table.update(subprocess_output)