python subprocess, multiple processes and redirection

Question

I spawn an amount of subprocesses and want to write the output to file and to console.

My current setup takes a dict of a file and a cmd which I want to process and then waits for it to finish, this works fine and the output is redirected to the file.

listofwork = [
    {'cmd':'somecmd1', 'file':'somefilename1.log'},
    {'cmd':'somecmd2', 'file':'somefilename2.log'},
    {'cmd':'somecmd3', 'file':'somefilename3.log'}
]
logpath = '/logs'

child_processes = []
for work in listofwork:
    with io.open(logpath + r'//' + work['file'],mode='wb') as out:
        p = subprocess.Popen(work['cmd'], stdout=out, stderr=out)
        child_processes.append(p)

for cp in child_processes:
    cp.communicate()

Basically I want to have this and also show the output onscreen. I have found this question.

Python Popen: Write to stdout AND log file simultaneously

but I want to keep the streams seperate so I can track which output is currently being read and act accordingly.

So How can I write to log and to screen without combining the into the stdout instantly. Most convenient would be If it would write to some internal data structure that I can manually print to screen! I need to be able to print in real-time and not wait for the process to end with .(communicate) as the process takes hours.

I can`t use StringIO because it throws an error that it has no attribute fileno()

how to write to log AND to screen without combining it in the subprocess.PIPE — , Jun 14 '14 at 14:30
You can use `StringIO` to capture the output in strings, then write it to both destinations. — Tom Zych, Jun 14 '14 at 14:33
you can`t use StringIO as subprocess expects to have real files and it ends with the error: 'StringIO instance has no attribute 'fileno'' — , Jun 14 '14 at 17:31
Why don't you want to use `subprocess.PIPE`? My recommendation would be to use pipes and then use [`select.select()`](https://docs.python.org/2/library/select.html#select.select) to wait until any of the pipes have data available to be read, at which point you then read from the pipe and then write that data to your stdout and to the corresponding log file. — Adam Rosenfield, Jun 14 '14 at 19:50
as far as I understand 'subprocess.PIPE' Doesn`t allow me differentiate between the commands I am running. It all throws it together into 1 single pipe — , Jun 14 '14 at 22:48
@RG337: Huh? Each subprocess you spawn with `stdout=subprocess.PIPE` and/or `stderr=subprocess.PIPE` gets its own separate pipe, I don't know where you got the idea that "it all throws it together into 1 single pipe." It becomes trickier to manage multiple pipes, since you have to avoid deadlock, but it can be done with careful use of `select()` or using multithreading. — Adam Rosenfield, Jun 15 '14 at 04:54
I see, It just seemed so obvious to me since they are called the same. I don`t really know how to select them yet as I have new to subprocess and have never heard of select(). I will report back on the results! Thanks — , Jun 15 '14 at 10:30

score 0 · Answer 1 · answered Jun 14 '14 at 19:18

The following cheats. Each subprocess now executes some work (ie: searching in a file), then uses the tee command to write to a file and to the subprocesses's stdout. Parent code takes each workers stdout and prints it to the screen. (It's simpler if a worker doesn't write to the screen directly -- messages would be intermixed.)

source

import multiprocessing, subprocess

listofwork = [
    {'cmd':'egrep whiskey recipe.txt', 'log':'booze.log'},
    {'cmd':'egrep -v whiskey recipe.txt', 'log':'other.log'},
]
# logpath = '/logs'

def worker(work):
    mylog = multiprocessing.get_logger()
    logged_cmd = '{cmd} | tee {log}'.format(**work)
    mylog.info('cmd: %s', logged_cmd)
    res = subprocess.check_output(logged_cmd, shell=True)
    mylog.info('res: %s', res)h
    return res


pool = multiprocessing.Pool()    # one process per CPU
for cmdout in pool.imap_unordered(worker, listofwork):
    print 'done:',cmdout

recipe.txt

whiskey
bitters
syrup

output

done: bitters
syrup

done: whiskey

python subprocess, multiple processes and redirection

1 Answers1

source

recipe.txt

output