multiprocessing and GUI updating - Qprocess or multiprocessing?

Question

After reading the literature on QProcesses and the multiprocessing module for python, I am still having trouble creating a working and responsive GUI throughout having large processes ongoing in the background. So far, I have come up with this simplified version of my application, which still shows similar problems to what many have described.

from PyQt4 import QtCore, QtGui
import multiprocessing as mp
import numpy as np
import sys
class Spectra:
    def __init__(self, spectra_name, X, Y):
        self.spectra_name = spectra_name
        self.X = X
        self.Y = Y
        self.iteration = 0

    def complex_processing_on_spectra(self, pipe_conn):
        self.iteration += 1
        pipe_conn.send(self.iteration)

class Spectra_Tab(QtGui.QTabWidget):
    def __init__(self, parent, spectra):
        self.parent = parent
        self.spectra = spectra
        QtGui.QTabWidget.__init__(self, parent)

        self.treeWidget = QtGui.QTreeWidget(self)
        self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
        self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])

        self.consumer, self.producer = mp.Pipe()
        # Make process associated with tab
        self.process = mp.Process(target=self.spectra.complex_processing_on_spectra, args=(self.producer,))

    def update_GUI(self, iteration):
        self.step.setText(1, str(iteration))

    def start_computation(self):
        self.process.start()
        while(True):
            message = self.consumer.recv()
            if message == 'done':
                break
            self.update_GUI(message)
        self.process.join()
        return

class MainWindow(QtGui.QMainWindow):
    def __init__(self, parent = None):
        QtGui.QMainWindow.__init__(self)

        self.setTabShape(QtGui.QTabWidget.Rounded)
        self.centralwidget = QtGui.QWidget(self)
        self.top_level_layout = QtGui.QGridLayout(self.centralwidget)

        self.tabWidget = QtGui.QTabWidget(self.centralwidget)
        self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25)

        process_button = QtGui.QPushButton("Process")
        self.top_level_layout.addWidget(process_button, 0, 1)
        QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process)

        self.setCentralWidget(self.centralwidget)
        self.centralwidget.setLayout(self.top_level_layout)

        # Open several files in loop from button - simplifed to one here
        X = np.arange(0.1200,.2)
        Y = np.arange(0.1200,.2)
        self.spectra = Spectra('name', X, Y)
        self.spectra_tab = Spectra_Tab(self.tabWidget, self.spectra)
        self.tabWidget.addTab(self.spectra_tab, 'name')

    def process(self):
        self.spectra_tab.start_computation()
        return

if __name__ == "__main__":
    app = QtGui.QApplication([])
    win = MainWindow()
    win.show()
    sys.exit(app.exec_())

This should be fully capable of executing if you have the dependencies. At the moment I have a QThreaded version of my program which works with signals and slots; Hwoever, I think it is important to have the ability to use all of a computers processors, since most users have ~8 cores available to them. So, I would like to expand this signal/slot threaded approach to the multiprocessed version using multiprocessing or QProcesses.
Does anyone have suggestions for whether or not to use QProcess or multiprocessing? While they are both complicated to me, QProcess seems as though it has less forums of people using pyQt, So I went with multiprocessing. Would it be simpler to go with QProcess since I already have signals/slots working with threads?

EDIT: Should I add a class like this as suggested?

class My_Process(QtCore.QProcess):
    def __init__(self, spectra):
        QtCore.QProcess.__init__(self)
        self.spectra = spectra

    def worker(self):
        QtConcurrent.run(self.spectra, self.spectra.complex_processing_on_spectra)

    def run(self):
        QtCore.QObject.connect(self, QtCore.SIGNAL(QTimer.timeout()), self.worker)

Lukas · Answer 1 · 2014-11-20T14:31:03.373

Even though the question is old and has been answered I would like add some clarification:

AFAIK QtConcurrent is not available in PyQt.
Threads in Qt with C++ are different from PyQt or python threads. The latter need to acquire python's GIL (global interpreter lock) when they run which effectively means that there is no real concurrency between python/pyqt threads.

Python threads are ok to keep the gui responsive, but you won't see any performance improvements for cpu bound tasks. I recommend using multiprocessing together with a QThread on your main process that handles the communication with the child process. You can use signals and slots between your main (gui) and your communication thread.

enter image description here

edit: I just had the same problem and did something along these lines:

from multiprocessing import Process, Queue
from PyQt4 import QtCore
from MyJob import job_function


# Runner lives on the runner thread

class Runner(QtCore.QObject):
    """
    Runs a job in a separate process and forwards messages from the job to the
    main thread through a pyqtSignal.

    """

    msg_from_job = QtCore.pyqtSignal(object)

    def __init__(self, start_signal):
        """
        :param start_signal: the pyqtSignal that starts the job

        """
        super(Runner, self).__init__()
        self.job_input = None
        start_signal.connect(self._run)

    def _run(self):
        queue = Queue()
        p = Process(target=job_function, args=(queue, self.job_input))
        p.start()
        while True:
            msg = queue.get()
            self.msg_from_job.emit(msg)
            if msg == 'done':
                break


# Things below live on the main thread

def run_job(input):
    """ Call this to start a new job """
    runner.job_input = input
    runner_thread.start()


def handle_msg(msg):
    print(msg)
    if msg == 'done':
        runner_thread.quit()
        runner_thread.wait()


# Setup the OQ listener thread and move the OQ runner object to it
runner_thread = QtCore.QThread()
runner = Runner(start_signal=runner_thread.started)
runner.msg_from_job.connect(handle_msg)
runner.moveToThread(runner_thread)

Did somebody notice a "shifting" when you have multiple arguments `args=(queue, inp1,inp2)` when I call a target function that is like: `def target_fun(self,inp1,inp2,...)` `Process` interprets this as `args=(queue, inp2,inp3)` what does the queue do here if i do not put it in the args everything runs fine... — www.pieronigro.de, Dec 17 '14 at 03:16
@Hiatus I hope you have solved your issue by now, however that looks like you are creating a process from a class member function, I'm not sure that you can do that? — Tom Myddeltyn, Jun 29 '16 at 17:52
Strangely, it seems that pattern like this works in PyQt, but does not work in PySide (1.2.4, on linux). In PySide, the `_run()` method connected to `runner_thread.started` still runs in the main thread. — kawing-chiu, Sep 23 '16 at 02:20
Is there any advantage in moving Runner to another thread instead of making it inherit QThread directly? — Buzz, Sep 03 '22 at 06:00

score 2 · Accepted Answer · answered Mar 28 '13 at 08:15

2

QProcess is for reading and writing to pipes (shell/cmd). Use one of these.

Create a class with worker function and run it as thread by QtConcurrent.run(object, method, args)
connect QTimer.timeout() with your worker function.

answered Mar 28 '13 at 08:15

Md. Minhazul Haque

619
5
14

1

Thank you very much for the help, but I am not sure about what you really mean. Should I make a class which inherits QProcess and spawn a thread from it? I have given an attempt at this type of class in my edited question. – chase Mar 28 '13 at 15:32
1

I am not good at python, doing Qt with C++ for years. But I understand a little of what you are trying to achieve. As far I know, 1. ```QProcess``` allowes you to start another process (say cmd.exe or firefox.exe or anything) inside your application and you can control (start, stop, write, read) them. 2. If you are trying to process (or do something) in background, please move to QThread or QTimer or QtConcurrent. All of these will run your piece of code in a separate process. So you can do one of the followings. – Md. Minhazul Haque Mar 28 '13 at 19:17
1

a. Use ```QThread```. Create a class that inherits QThread and overload its run() method. Write your complex_processing_on_spectra methods in it. Or you can use Spectra object inside that class. Emit signal when an iteration takes place. Connect this signal with a slot that will update the UI. b. Use ```QTimer```. Connect timer's ```timeout()``` signal with ```update_GUI()``` with a interval of n milisec. c. Use ```QtConcurrent```. But this time you can avoid it. I am not sure if it is a solid API. – Md. Minhazul Haque Mar 28 '13 at 19:17
Thanks @Muhammad I am now seeing what you mean by `QProcess` will not run since it only runs executables. However, is it not true that `QThreads` and multithreading only will use one processor on the CPU? Therefore, when you have a program that is only multi-threaded it may only be able to use ~15% of your processor if you have a modern computer (~8 cpu cores)? – chase Mar 29 '13 at 01:21
You can use more CPU by using threads. If your application has 2 thread, system policy will let you use 2 cores. Actually ```multi-core processing``` is a fact how you will manage threads. Qt supports ```moving threads into``` another. So you can implement an algorithm on how to manage threads, arrange them etc. – Md. Minhazul Haque Mar 29 '13 at 23:11
Just to point that multithreading in C++ isn't equivalent to multithreading in Python. In Python we have the GIL that limits the app to 1 CPU. Multiprocessing is required in Python if we want to use more CPUs. – chbp Mar 21 '22 at 17:37

multiprocessing and GUI updating - Qprocess or multiprocessing?

2 Answers2

Linked