0

I'm trying to run an unknown number of commands and capture their stdout in a file. However, I am presented with a difficulty when attempting to p.wait() on each instance. My code looks something like this:

print "Started..."
for i, cmd in enumerate(commands):
    i = "output_%d.log" % i
    p = Popen(cmd, shell=True, universal_newlines=True, stdout=open(i, 'w'))
    p.wait()
print "Done!"

I'm looking for a way to execute everything in commands simultaneously and exit the current script only when each and every single process has been completed. It would also help to be informed when each command returns an exit code.

I've looked at some answers, including this one by J.F. Sebastian and tried to adapt it to my situation by changing args=(p.stdout, q) to args=(p.returncode, q) but it ended up exiting immediately and running in the background (possibly due to shell=True?), as well as not responding to any keys pressed inside the bash shell... I don't know where to go with this.

Jeremy Brown's answer also helped, sort of, but select.epoll() was throwing an AttributeError exception.

Is there any other seamless way or trick to make it work? It doesn't need to be cross platform, a solution for GNU/Linux and macOS would be much appreciated. Thanks in advance!

Community
  • 1
  • 1
pwn'cat
  • 111
  • 1
  • 6

2 Answers2

0

A big thanks to Adam Matan for the biggest hint towards the solution. This is what I came up with, and it works flawlessly:

  1. It initiates each Thread object in parallel
  2. It starts each instance simultaneously
  3. Finally it waits for each exit code without blocking other threads

Here is the code:

import threading
import subprocess

...

def run(cmd):
    name = cmd.split()[0]
    out = open("%s_log.txt" % name, 'w')
    err = open('/dev/null', 'w')
    p = subprocess.Popen(cmd.split(), stdout=out, stderr=err)
    p.wait()
    print name + " completed, return code: " + str(p.returncode)

...

proc = [threading.Thread(target=run, args=(cmd)) for cmd in commands]
[p.start() for p in proc]
[p.join() for p in proc]
print "Done!"
Community
  • 1
  • 1
pwn'cat
  • 111
  • 1
  • 6
0

I would have rathered add this as a comment because I was working off of Jack of all Spades' answer. I had trouble getting that exact command to work because it was unpacking the string list I had of commands.

Here's my edit for python3:

import subprocess
import threading

commands = ['sleep 2', 'sleep 4', 'sleep 8']

def run(cmd):
    print("Command %s" % cmd)
    name = cmd.split(' ')[0]
    print("name %s" % name)
    out = open('/tmp/%s_log.txt' % name, 'w')
    err = open('/dev/null', 'w')
    p = subprocess.Popen(cmd.split(' '), stdout=out, stderr=err)
    p.wait()
    print(name + " completed, return code: " + str(p.returncode))


proc = [threading.Thread(target=run, kwargs={'cmd':cmd}) for cmd in commands]
[p.start() for p in proc]
[p.join() for p in proc]
print("Done!")
Dan Corbiani
  • 11
  • 3