python parallel processes return exit code

Question

lets see if I can make this clear... I'm a total Python beginner so bear with me, this is my first python program (though I'm familiar with basic scripting in a few other languages). I've been searching around for hours and I'm sure the answer to this is fairly simple but I have yet to get it to work properly.

I'm writing a code that should launch multiple commandline processes, and when each one finishes I want to update a cell in a QTableWidget. The table has a row for each process to run, and each row has a cell for the "status" of the process.

I can run this no problem if I just do a for loop, spawning one process per row using subprocess.call() however this is too linear and I would like to fire them all off at the same time and not hang the program for each loop cycle. I've been digging through the subprocess documentation and am having a really hard time with it. I understand that I need to use subprocess.Popen (which will prevent my program from hanging while the process runs, and thus I can spawn multiple instances). Where I run into trouble is getting the exit code back so that I can update my table, without hanging the program - for instance using subprocess.wait() followed by a subprocess.returncode still just sticks until the process completes. I need a sort of "when process completes, check the exit code and run a function that updates the QTableWidget."

I did find these two posts that seemed to get me going in the right direction, but didn't quite get me there:

Understanding Popen.communicate

How to get exit code when using Python subprocess communicate method?

Hopefully that made sense. Here's a simplified version of my code, I realize it is half-baked and half-broken but I've been screwing around with it for over an hour and I've lost track of a few things...

import os, subprocess

ae_app = 'afterfx'
ae_path = os.path.join('C:/Program Files/Adobe/Adobe After Effects CC 2015/Support Files', ae_app + ".exe")
filename = "E:/Programming/Python/Archive tool/talk.jsx"
commandLine = 'afterfx -noui -r ' + filename

processList = [commandLine]
processes = []

for process in processList:
    f = os.tmpfile()
    aeProcess = subprocess.Popen(process, executable=ae_path, stdout=f)
    processes.append((aeProcess, f))

for aeProcess, f in processes:
    # this is where I need serious help...
    aeProcess.wait()
    print "the line is:"
    print aeProcess.returncode

• Spencer

Answer 1

You mentioned PyQt, so you can use PyQt's QProcess class.

def start_processes(self, process_list):
    for cmd, args in process_list:
        proc = QProcess(self)
        proc.finished.connect(self.process_finished)
        proc.start(cmd, args)

def process_finished(self, code, status):
    # Do something

UPDATE: Added fully working example. Works properly for both PyQt4 and PyQt5 (to switch just comment line 3 and uncomment line 4)

sleeper.py

import sys
from time import sleep
from datetime import datetime as dt

if __name__ == '__main__':
    x = int(sys.argv[1])
    started = dt.now().time()
    sleep(x)
    ended = dt.now().time()
    print('Slept for: {}, started: {}, ended: {}'.format(x, started, ended))
    sys.exit(0)

main.py

import sys

from PyQt5 import QtCore, QtWidgets
# from PyQt4 import QtCore, QtGui as QtWidgets


class App(QtWidgets.QMainWindow):
    cmd = r'python.exe C:\_work\test\sleeper.py {}'

    def __init__(self):
        super(App, self).__init__()
        self.setGeometry(200, 200, 500, 300)
        self.button = QtWidgets.QPushButton('Start processes', self)
        self.button.move(20, 20)
        self.editor = QtWidgets.QTextEdit(self)
        self.editor.setGeometry(20, 60, 460, 200)

        self.button.clicked.connect(self.start_proc)

    def start_proc(self):
        for x in range(5):
            proc = QtCore.QProcess(self)
            proc.finished.connect(self.finished)
            proc.start(self.cmd.format(x))

    def finished(self, code, status):
        self.editor.append(str(self.sender().readAllStandardOutput()))


if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    gui = App()
    gui.show()
    app.exec_()