Python Subprocess: replicating Tee with subprocess.Popen and executing in parallel?

Question

I am trying to write a python script to compile and upload in parallel the same hex file to multiple microcontrollers via Arduino's command line interface.

My script does the following:

Compile the ino file to a hex file in a particular directory. For example,
Upload the hex to all /dev/tty.usbXXXXXX.

These are the requirements:

I would have to be able to upload to multiple /dev/tty.usb* in parallel.
I want to print all the stdout and stderr from all my subprocess.Popen into my main screen with DEVICE - STDOUT/STDERR - MSG as the format
I want to save both stdout and stderr from each Popen into its own tty.usb* log file.

Right now i have:

import errno
import os
import re
import subprocess

ARDUINO_EXECUTABLE = '/Applications/Arduino.app/Contents/MacOS/JavaApplicationStub'
HEX_FOLDER_DIR = '/tmp/oyoroi'
LOG_FOLDER_DIR = './logs'


def get_devices():
  """Returns a list of tty.usbXXXXXX
  Make sure you use the USB hub. This will force an extra character in the /dev/tty.usbXXXXXX
  """
  ret = []
  for device in os.listdir('/dev'):
    if re.match('tty.usbmodem[0-9]{6}', device):
      ret.append(device)
  return ret


class Wrapper(object):
  """Wrapper for file objects
  """
  def __init__(self, name, fobject):
    self.name = name
    self.fobject = fobject

  def fileno(self):
    return self.fobject.fileno()

  def flush(self):
    self.fobject.flush()

  def write(self, a_str):
    print self.name, a_str
    self.fobject.write(a_str)


def main(fname):
  """Build once, but upload in parallel
  """
  try:
    os.makedirs(HEX_FOLDER_DIR)
  except OSError as exc:
    if exc.errno == errno.EEXIST and os.path.isdir(HEX_FOLDER_DIR):
      pass

  fname = os.path.abspath(fname)

  # Builds the hex
  command = "%s --verify --pref build.path=%s %s" % (ARDUINO_EXECUTABLE, HEX_FOLDER_DIR, fname)
  print "(Build Command)", command
  proc = subprocess.call(command, shell=True)

  # Make the log directory
  try:
    os.makedirs(LOG_FOLDER_DIR)
  except OSError as exc:
    if exc.errno == errno.EEXIST and os.path.isdir(LOG_FOLDER_DIR):
      # delete folder
      import shutil
      shutil.rmtree(LOG_FOLDER_DIR)
      # and recreate again
      os.makedirs(LOG_FOLDER_DIR)

  # Upload in parallel
  devices = get_devices()
  processes = []

  for device in devices:
    device_path = '/dev/' + device
    log_file_path = os.path.join(LOG_FOLDER_DIR, device + '.log')
    with open(log_file_path, 'a') as logfile:
      command = "%s --upload --pref build.path=%s --port %s %s" % \
                (ARDUINO_EXECUTABLE, HEX_FOLDER_DIR, device_path, fname)
      print "(Upload Command)", command

      wstdout = Wrapper('%_STDOUT', logfile)
      wstderr = Wrapper('%_STDERR', logfile)
      proc = subprocess.Popen(command, shell=True, stdout=wstdout, stderr=wstderr)
      processes.append(proc)


if __name__ == "__main__":
  import sys
  if len(sys.argv) != 2:
    print "python upload.py <.ino>"
    exit(1)
  main(sys.argv[1])

I am able to get what I want in each of the log files, but my terminal is not printing anything on the screen. It's also ending before the other processes are complete.

What am I missing?

`stdout=wstdout` is incorrect. `subprocess.Popen()` doesn't accept a file-like object (all Wrapper methods except `.fileno()` are ignored in your case that is why you don't see anything -- nothing is printed (`print self.name, a_str` is not called)). It needs a real file (real file descriptor). See [how `teed_call()` that uses multiple threads is implemented](http://stackoverflow.com/a/4985080/4279). Here's [how to do collect both stdout/stderr independently and print it to console in a single thread](http://stackoverflow.com/a/25960956/4279). — jfs, Oct 29 '14 at 16:07

score 0 · Answer 1 · answered Oct 29 '14 at 02:17

0

At the end of your main(), add code like this:

for proc in processes:
    proc.wait()

Right now, you are not waiting, so Python is exiting as soon as all the processes are launched.

For the record, I'm not entirely certain how useful it is to pass Wrapper objects instead of true file objects. Python is very likely just calling .fileno() and then attaching the subprocesses to that file descriptor directly. Your .write() method is probably not getting called. If you need it to be called, you should use subprocess.PIPE and Popen.communicate() in place of the Wrapper objects and Popen.wait(). You can then use the return value of communicate() in whatever way you see fit (i.e., print it).

answered Oct 29 '14 at 02:17

Kevin

28,963
9
62
81

So i tried doing something like this: https://gist.github.com/vicngtor/40d9cb41780a0d5f14e8 Nothing gets printed at the end. Am I missing something? – Tinker Oct 29 '14 at 02:31
`communicate()` only works if you pass `subprocess.PIPE` as `stdout` and/or `stderr` (and then only for the stream(s) you passed). In this case, I believe you will need to both `write()` to the log files and `print()` to the screen by hand, using the `communicate()` return value. – Kevin Oct 29 '14 at 02:37

Python Subprocess: replicating Tee with subprocess.Popen and executing in parallel?

1 Answers1