0

My problem is that I have a list of shell commands that need to be executed serially from a python program UI. The shell commands can require anywhere from 10 seconds to 10 minutes to finish. And I want them run in the background without blocking the main UI/thread so I continue to use the python program. I have tried the following to run a number of shell commands.

command_list = list()
for i in command_list:
    os.system("Running command: ", i)

os.system() will execute the command list one by one, but will block the main thread The same apply for subprocess.run() andsubprocess.call()

subprocess.Popen() will not block the main thread but will run all the shell commands in parallel which is not desirable.

I have tried googling and asking in python discord for the past few days but haven't been able find a solution to my problem.

Edit: Supposedly I have this GUI script, just a QMainWindow with two QPushButton ("Start", "Stop"). "Start" will run a list of shell commands. "Stop" will stop the execution of the shell commands.

class Window(qt.QMainWindow):
    def __init__(self):
    super().__init__()
    central_widget = qt.QWidget()
    central_widget.setLayout(qt.QHBoxLayout())
    self.setCentralWidget(central_widget)

    centra_widget.layout().addWidget(qt.QPushButton("Start", clicked = self.run_command))
    centra_widget.layout().addWidget(qt.QPushButton("Stop", clicked = self.stop_command))

    def run_command(self):
        for cmd in cmd_list:
            subprocess.Popen("Run command", cmd)

    def stop_command(self):
        # Do something to stop the shell commands

app = qt.QApplication([])
main_window = Window()
main_window.show()
app.exec_()

The code above will run every commands in the list at the same time.

So i tried putting sleep() between the Popen but it will also freeze the main UI and the "Stop" button cannot be clicked.

That was my original question as how to run shell commands in order and not freeze the main UI/thread. Thanks for you time.

Link Phan
  • 1
  • 2
  • Does this answer your question? [PyQt5 and subprocess.Popen(...)](https://stackoverflow.com/questions/51828943/pyqt5-and-subprocess-popen) – tripleee Jul 06 '21 at 08:24

1 Answers1

0

subprocess.Popen only starts a single process. Your calling program will need to poll it from time to time to figure out if it has finished, and only then run the next command.

import subprocess
import shlex
import time

# ...

for cmd in command_list:
    print('running command:', cmd)
    p = subprocess.Popen(shlex.split(cmd))
    result = None
    while result is None:
        time.sleep(1) # or whatever else you want to do here
        result = p.poll()
    if result != 0:
        raise CalledProcessError(
            '%s failed, result code %i', cmd, result)

If your progam is governed by a GUI main loop, you will probably want to turn this inside out, as it were: Ask the main loop to run a callback, say, every second, and in the callback do the poll and possible spawning of a new process when the previous one is done.

If you are running threading for other reasons anyway, you could just loop over your command list and run each with subprocess.check_call() or similar in a dedicated thread instead.

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • I suppose you could run all of them via `multiprocessing.Pool(processes=1)` which creates a pool with a single process. Then the `multiprocessing` library will take care of the plumbing to wait for one process to finish and the next one to start. – tripleee Jul 04 '21 at 16:39
  • Thank you for your hint. Can you show me a code example? – Link Phan Jul 04 '21 at 23:10
  • The [`subprocess` documentation](https://docs.python.org/3/library/subprocess.html) has many simple examples; you basically want to `Popen` and then periodically `poll` the `Popen` object. Your question doesn't contain enough information about the calling code to allow us to guess what should happen between `Popen` and `poll` so this is necessarily vague. – tripleee Jul 05 '21 at 06:13
  • Thank you for your time. I have updated the code to provide more concrete information about my use-case. Hope it clears it up. – Link Phan Jul 05 '21 at 21:32
  • Thanks for the update. I'm not familiar with your GUI framework but quick duck duck going gets me https://stackoverflow.com/questions/51828943/pyqt5-and-subprocess-popen – tripleee Jul 06 '21 at 08:17