0

I have an multiprocess GUI application that works flawlessly when invoked using python on Mac.
For spawning my processes, and running the function func asynchronously, I'm using multiprocessing.pools:

def worker(self): 
        calulation_promise = self._pool.map_async(func, (self.simulated_values), 
                        callback=self.simFin, error_callback=self.simError)
        return calulation_promise

Now, I need to make an executable from my project using cx-freeze. I'm using the documentation's template provided here.

import sys
from cx_Freeze import setup, Executable

# Dependencies are automatically detected, but it might need fine tuning.
# "packages": ["os"] is used as example only
excludes = ["Pyside2.Qt5WebEngineCore.dll", "PySide6"]
build_exe_options = {"packages": ['multiprocessing'], "excludes": excludes}


# base="Win32GUI" should be used only for Windows GUI app
base = None
if sys.platform == "win32":
    base = "Win32GUI"

setup(
    name = "guifoo",
    version = "0.1",
    description = "My GUI application!",
    options = {"build_exe": build_exe_options},
    executables = [Executable("main.py", base=base)]
)

Unfortunately, when my main application calls my worker () function, it always starts the main process (which starts a new MainWindow GUI). That means, instead of executing the function func, it somehow starts the main thread over and over again (see the outputs for more clarity)

Working example:

Note: This is a working example to reproduce the issue.

import sys, os, time, logging, platform, multiprocessing, random
from multiprocessing import Process, Pool, cpu_count, freeze_support
from PySide2.QtWidgets import (QLineEdit, QPushButton, QApplication, QVBoxLayout, QDialog)
from PySide2.QtCore import Signal, QObject
from rich.logging import RichHandler

class Form(QDialog):

    def __init__(self, parent=None):
        super(Form, self).__init__(parent)
        logging.info ("Stared gui...")
        # Create widgets
        self.edit = QLineEdit("<empty>")
        self.button = QPushButton("Start worker")
        # Create layout and add widgets
        layout = QVBoxLayout()
        layout.addWidget(self.edit)
        layout.addWidget(self.button)
        self.setLayout(layout)                          # Set dialog layout
        self.button.clicked.connect(self.startWorker)   # Add button signal to greetings slot

    # Greets the user
    def startWorker(self):
        logging.info("Stared worker...")
        tw = ThreadWrapper()
        self.promise = tw.worker()
        tw.signals.finished.connect(self.finished)
    
    def finished(self):
        self.edit.setText(str(self.promise.get()))

class ThreadWrapper():
    def __init__(self):
        self.simulated_values = range(1, 30, 1)
        self._pool = Pool(processes=8)
        self.signals = WorkerSignals()

    def simFin(self, value):
        logging.info("%s" % (value))
        self.signals.finished.emit()
    
    def simError(self, value):
        logging.error("%s" % (value))

    def worker(self):
        
        calulation_promise = self._pool.map_async(func, (self.simulated_values), 
                        callback=self.simFin, error_callback=self.simError)
        return calulation_promise


class WorkerSignals(QObject):
    finished = Signal()

# A function which needs an arbitrary amount of time to finish
def func(value):
    wait = random.randint(1, 5); time.sleep(wait) 
    res = value**value
    print("THREAD: %d*%d = %d; waiting %d" % (value, value, res, wait))
    return res

def main():
    logging.basicConfig(level="DEBUG", format="%(name)s | %(message)s", datefmt="[%X]", handlers=[RichHandler()])
    if platform.system() == "Darwin":
            multiprocessing.set_start_method('spawn')
            os.environ['QT_MAC_WANTS_LAYER'] = '1'
            os.environ['QT_MAC_USE_NSWINDOW'] = '1'
            
    app = QApplication(sys.argv)
    window = Form()
    window.show()  
    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

Outputs

Using python

[20:41:50] INFO     root | Stared gui...                             _main.py:11
[20:41:52] INFO     root | Stared worker...                          _main.py:24
THREAD: 3*3 = 27; waiting 1
THREAD: 4*4 = 256; waiting 3
THREAD: 1*1 = 1; waiting 5
THREAD: 2*2 = 4; waiting 5
[20:41:57] INFO     root | [1, 4, 27, 256]   

Using the executable

[20:44:03] INFO     root | Stared gui...                             _main.py:11
[20:44:05] INFO     root | Stared worker...                          _main.py:24
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11
[20:44:06] INFO     root | Stared gui...                             _main.py:11

Additional Details:

  • Mac OS Big Sur, Ver. 11.5.1
  • Python 3.7.4
  • PySide2, ver 5.15.2
agentsmith
  • 1,226
  • 1
  • 14
  • 27
  • You probably should call `freeze_support` in `main`. You import it but never actually called it. – Aaron Oct 28 '21 at 19:30
  • 1
    According to [the official documentation](https://docs.python.org/3/library/multiprocessing.html) it is only valid for Windows executables: *Calling freeze_support() has no effect when invoked on any operating system other than Windows.* Therefore I have not included it. – agentsmith Oct 28 '21 at 19:39
  • I understand that's the case, but the behavior you describe is analogous to what `freeze_support` seeks to solve. I'm not familiar with cx-freeze on MacOS, so that may not be the solution, but it's worth a try no? Python has semi-recently started to default to "spawn" on MacOS, which is more similar to how windows always has to work. – Aaron Oct 28 '21 at 19:42
  • Yes, I get your point and actually I have tried it in my application where this problem occurred in the first place, but it did not change anything. – agentsmith Oct 28 '21 at 19:46
  • nvm.. : [*"Warning The 'spawn' and 'forkserver' start methods cannot currently be used with “frozen” executables (i.e., binaries produced by packages like PyInstaller and cx_Freeze) on Unix. The 'fork' start method does work."*](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) TLDR; use "fork" instead of spawn – Aaron Oct 28 '21 at 19:48
  • 1
    though... it should already be using "fork" because "spawn" was a 3.8 update.... IDK worth trying `mp.set_start_method('fork')` – Aaron Oct 28 '21 at 19:49
  • I would also update python to 3.9. multiprocessing has gotten some much needed tlc in the past few years in terms of bugfixes. 3.10 is still a little too fresh out of the oven... – Aaron Oct 28 '21 at 20:13
  • Yeah, `fork` solved the problem in the given minimal example above, but my program now exits with the following message: `The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec()` - see https://stackoverflow.com/questions/30669659/multiproccesing-and-error-the-process-has-forked-and-you-cannot-use-this-corefou – agentsmith Oct 29 '21 at 10:38
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/238699/discussion-between-aaron-and-agentsmith). – Aaron Oct 29 '21 at 12:47

0 Answers0