When I transform my script in a standalone executable using Pyinstaller and try to run a CrawlerRunner process that I got from this answer (which is connected to the Update Database button) my GUI opens a new window and completely freezes. When running in script form it works just fine. Like this:
In main.py:
class MyGUI(QMainWindow):
def __init__(self):
...
self.update_db_button.clicked.connect(self.on_clicked_update)
...
def on_clicked_update(self):
"""Gives command to run scraper and fetch data from the website"""
run_spider()
gui_warnings.update_notification()
In scrapy_run.py
from multiprocessing import Queue
from multiprocessing import Process
from twisted.internet import reactor
from scrapy import crawler
from currency_scraper.currency_scraper.spiders.investor import InvestorSpider
# from https://stackoverflow.com/a/43661172/13825145
def f(q):
try:
runner = crawler.CrawlerRunner(
{
"USER_AGENT": "currency_scraper",
"SCRAPY_SETTINGS_MODULE": "currency_scraper.currency_scraper.settings",
"ITEM_PIPELINES": {
"currency_scraper.currency_scraper.pipelines.Sqlite3Pipeline": 300
}
}
)
deferred = runner.crawl(InvestorSpider)
deferred.addBoth(lambda _: reactor.stop())
reactor.run()
q.put(None)
except Exception as e:
q.put(e)
def run_spider():
q = Queue()
p = Process(target=f, args=(q,))
p.start()
result = q.get()
p.join()
if result is not None:
raise result
Console log
[9004] PyInstaller Bootloader 3.x
[9004] LOADER: executable is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist\Currency Converter.exe
[9004] LOADER: homepath is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist
[9004] LOADER: _MEIPASS2 is NULL
[9004] LOADER: archivename is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist\Currency Converter.exe
[9004] LOADER: Extracting binaries
[9004] LOADER: Executing self as child
[9004] LOADER: set _MEIPASS2 to C:\Users\Familia\AppData\Local\Temp\_MEI90042
[9004] LOADER: Setting up to run child
[9004] LOADER: Creating child process
[9004] LOADER: Waiting for child process to finish...
[240] PyInstaller Bootloader 3.x
[240] LOADER: executable is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist\Currency Converter.exe
[240] LOADER: homepath is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist
[240] LOADER: _MEIPASS2 is C:\Users\Familia\AppData\Local\Temp\_MEI90042
[240] LOADER: archivename is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist\Currency Converter.exe
[240] LOADER: SetDllDirectory(C:\Users\Familia\AppData\Local\Temp\_MEI90042)
[240] LOADER: Already in the child - running user's code.
[240] LOADER: manifestpath: C:\Users\Familia\AppData\Local\Temp\_MEI90042\Currency Converter.exe.manifest
[240] LOADER: Activation context created
[240] LOADER: Activation context activated
[240] LOADER: Python library: C:\Users\Familia\AppData\Local\Temp\_MEI90042\python38.dll
[240] LOADER: Loaded functions from Python library.
[240] LOADER: Manipulating environment (sys.path, sys.prefix)
[240] LOADER: sys.prefix is C:\Users\Familia\AppData\Local\Temp\_MEI90042
[240] LOADER: Pre-init sys.path is C:\Users\Familia\AppData\Local\Temp\_MEI90042\base_library.zip;C:\Users\Familia\AppData\Local\Temp\_MEI90042
[240] LOADER: Setting runtime options
[240] LOADER: Bootloader option: pyi-windows-manifest-filename Currency Converter.exe.manifest
[240] LOADER: Initializing python
[240] LOADER: Overriding Python's sys.path
[240] LOADER: Post-init sys.path is C:\Users\Familia\AppData\Local\Temp\_MEI90042\base_library.zip;C:\Users\Familia\AppData\Local\Temp\_MEI90042
[240] LOADER: Setting sys.argv
[240] LOADER: setting sys._MEIPASS
[240] LOADER: importing modules from CArchive
[240] LOADER: extracted struct
[240] LOADER: callfunction returned...
[240] LOADER: extracted pyimod01_os_path
[240] LOADER: callfunction returned...
[240] LOADER: extracted pyimod02_archive
[240] LOADER: callfunction returned...
[240] LOADER: extracted pyimod03_importers
[240] LOADER: callfunction returned...
[240] LOADER: Installing PYZ archive with Python modules.
[240] LOADER: PYZ archive: PYZ-00.pyz
[240] LOADER: Running pyiboot01_bootstrap.py
[240] LOADER: Running pyi_rth__tkinter.py
[240] LOADER: Running pyi_rth_multiprocessing.py
[240] LOADER: Running pyi_rth_pkgres.py
[240] LOADER: Running pyi_rth_mplconfig.py
[240] LOADER: Running pyi_rth_mpldata.py
[240] LOADER: Running pyi_rth_twisted.py
[240] LOADER: Running pyi_rth_pyqt5.py
[240] LOADER: Running main.py
c:\users\familia\documents\programação\python\projetos\currency_converter\venv\lib\site-packages\PyInstaller\loader\pyimod03_importers.py:493: MatplotlibDeprecationWarning:
The MATPLOTLIBDATA environment variable was deprecated in Matplotlib 3.1 and will be removed in 3.3.
exec(bytecode, module.__dict__)
[14188] PyInstaller Bootloader 3.x
[14188] LOADER: executable is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist\Currency Converter.exe
[14188] LOADER: homepath is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist
[14188] LOADER: _MEIPASS2 is C:\Users\Familia\AppData\Local\Temp\_MEI90042
[14188] LOADER: archivename is C:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\dist\Currency Converter.exe
[14188] LOADER: SetDllDirectory(C:\Users\Familia\AppData\Local\Temp\_MEI90042)
[14188] LOADER: Already in the child - running user's code.
[14188] LOADER: manifestpath: C:\Users\Familia\AppData\Local\Temp\_MEI90042\Currency Converter.exe.manifest
[14188] LOADER: Activation context created
[14188] LOADER: Activation context activated
[14188] LOADER: Python library: C:\Users\Familia\AppData\Local\Temp\_MEI90042\python38.dll
[14188] LOADER: Loaded functions from Python library.
[14188] LOADER: Manipulating environment (sys.path, sys.prefix)
[14188] LOADER: sys.prefix is C:\Users\Familia\AppData\Local\Temp\_MEI90042
[14188] LOADER: Pre-init sys.path is C:\Users\Familia\AppData\Local\Temp\_MEI90042\base_library.zip;C:\Users\Familia\AppData\Local\Temp\_MEI90042
[14188] LOADER: Setting runtime options
[14188] LOADER: Bootloader option: pyi-windows-manifest-filename Currency Converter.exe.manifest
[14188] LOADER: Initializing python
[14188] LOADER: Overriding Python's sys.path
[14188] LOADER: Post-init sys.path is C:\Users\Familia\AppData\Local\Temp\_MEI90042\base_library.zip;C:\Users\Familia\AppData\Local\Temp\_MEI90042
[14188] LOADER: Setting sys.argv
[14188] LOADER: setting sys._MEIPASS
[14188] LOADER: importing modules from CArchive
[14188] LOADER: extracted struct
[14188] LOADER: callfunction returned...
[14188] LOADER: extracted pyimod01_os_path
[14188] LOADER: callfunction returned...
[14188] LOADER: extracted pyimod02_archive
[14188] LOADER: callfunction returned...
[14188] LOADER: extracted pyimod03_importers
[14188] LOADER: callfunction returned...
[14188] LOADER: Installing PYZ archive with Python modules.
[14188] LOADER: PYZ archive: PYZ-00.pyz
[14188] LOADER: Running pyiboot01_bootstrap.py
[14188] LOADER: Running pyi_rth__tkinter.py
[14188] LOADER: Running pyi_rth_multiprocessing.py
[14188] LOADER: Running pyi_rth_pkgres.py
[14188] LOADER: Running pyi_rth_mplconfig.py
[14188] LOADER: Running pyi_rth_mpldata.py
[14188] LOADER: Running pyi_rth_twisted.py
[14188] LOADER: Running pyi_rth_pyqt5.py
[14188] LOADER: Running main.py
c:\users\familia\documents\programação\python\projetos\currency_converter\venv\lib\site-packages\PyInstaller\loader\pyimod03_importers.py:493: MatplotlibDeprecationWarning:
The MATPLOTLIBDATA environment variable was deprecated in Matplotlib 3.1 and will be removed in 3.3.
exec(bytecode, module.__dict__)
