Run multiple subprocesses in parallel displaying all output on screen until complete

Question

I'm learning python and could use some help. The following code executes 4 subprocess commands. Each command is executed one after the next with all of it's output and errors being displayed. How to I preserve the behavior of waiting and displaying all output, but execute multiple commands concurrently? I've looked at running several system commands in parallel which seems to approximate what I think I'm going for, but am not sure how to make it work and maintain the output to stdout for each process.

def execute(command):
  process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  while True:
      nextline = process.stdout.readline()
      if nextline == '' and process.poll() != None:
          break
      sys.stdout.write(nextline)
      sys.stdout.flush()

  output = process.communicate()[0]
  returncode = process.returncode

  if (returncode == 0):
      return output
  else:
    raise Exception("%s %s %s" % (command, returncode, output))

execute(cmd1)
execute(cmd2)
execute(cmd3)
execute(cmd4)

score 3 · Answer 1 · answered Jul 02 '15 at 15:53

Looks to me like your function may be waiting because of this part:

  while True:
      nextline = process.stdout.readline()
      if nextline == '' and process.poll() != None:
          break
      sys.stdout.write(nextline)
      sys.stdout.flush()

  output = process.communicate()[0]

You should call the Popen() on all commands before calling communicate() on the first.

eg.

a = Popen(cmd, stdout=PIPE, stderr=PIPE, shell=True)
b = Popen(cmd, stdout=PIPE, stderr=PIPE, shell=True)
c = Popen(cmd, stdout=PIPE, stderr=PIPE, shell=True)
# wait for them to finish
astdout, astderr = a.communicate()
bstdout, bstderr = b.communicate()
cstdout, cstderr = c.communicate()

print astdout
print bstdout
print cstdout

Run multiple subprocesses in parallel displaying all output on screen until complete

1 Answers1