How to add dynamic python modules to PyInstaller's specs?

Question

I'm trying to figure out how to load dynamic/hidden imports with PyInstaller, so far I got this simple structure:

First of all, I got a framework package added into my PYTHONPATH living in d:\Sources\personal\python\framework

That package is used by many of my python projects, in particular, it's used with the below simple project I want to package

Main project

├───data        <- Pure static data
├───plugins     <- Dynamic modules which uses framework's modules                                      
├───resources   <- Static data+embedded (generated by pyqt), used by plugins
│   ├───css                                              
│   ├───images                    
|   resources.py
|   resources.qrc                       
main.py         <- Uses framework's modules to load plugins dynamically

My spec file looks like this:

# -*- mode: python -*-

block_cipher = None


a = Analysis(['main.py'],
             pathex=['d:\\sources\\personal\\python\\pyqt\\pyshaders'],
             binaries=None,
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)

##### include mydir in distribution #######
def extra_datas(mydir):
    def rec_glob(p, files):
        import os
        import glob
        for d in glob.glob(p):
            if os.path.isfile(d):
                files.append(d)
            rec_glob("%s/*" % d, files)
    files = []
    rec_glob("%s/*" % mydir, files)
    extra_datas = []
    for f in files:
        extra_datas.append((f, f, 'DATA'))

    return extra_datas
###########################################

a.datas += extra_datas('data')

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='main',
          debug=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='main')

The problem comes when I try to add hidden imports to the Analysis(... , hiddenimports=[], ...)'s hiddenimports list, I've tried so far this:

hiddenimports=['d:\\sources\\personal\\python\\pyqt\\plugins']
hiddenimports=['d:\\sources\\personal\\python\\pyqt\\plugins\\*']
hiddenimports=['plugins']

Also tried listing as individual files with absolute paths:

hiddenimports=[
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\api.py',
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\config.py',
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\plugins_actions.py',
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\plugins_dialogs.py',
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\plugins_docks.py',
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\plugins_post_init.py',
    'd:\\sources\\personal\\python\\pyqt\\pyshaders\\plugins\\plugins_toolbar.py'
]

And also tried to load them as module packages (__init__.py lives in plugins folder)

hiddenimports=[
    'plugins.api',
    'plugins.config',
    'plugins.plugins_actions',
    'plugins.plugins_dialogs',
    'plugins.plugins_docks',
    'plugins.plugins_post_init',
    'plugins.plugins_toolbar'
]

Also tried the collect_submodules

hiddenimports=collect_submodules('plugins')

None of these attempts worked and the folder plugin isn't being added properly to the dist (when i say 'properly' i guess pyinstaller will analize the imports used by these hidden plugins analizing recursively their dependencies and copying the *.pyc files)... So, I'd like to know how can i add properly "hidden" modules which are being loaded dynamically to the pyinstaller's spec.

You could try adding the files in the [`pure`](https://pythonhosted.org/PyInstaller/spec-files.html#spec-file-operation) dependencies by using [`TOC`](https://pythonhosted.org/PyInstaller/advanced-topics.html#toc-class-table-of-contents). — Repiklis, Nov 10 '16 at 10:05
Possibly hitting this (https://github.com/pyinstaller/pyinstaller/issues/2009) issue? — Peter Brittain, Nov 13 '16 at 11:15
Did you try using PyInstaller hooks (https://pyinstaller.readthedocs.io/en/stable/hooks.html)? They are designed exactly for that purpose. — void, Nov 16 '16 at 15:46
@void Not really, although I've come up with a suboptimal solution where just copying the plugins in the final dist and marking them as hiddenimports will do, this solution forces you to expose python code to the final user though. — BPL, Nov 16 '16 at 15:52
You don't say how you load these plugins in your Python code. That affects what you give to `hiddenimports`. Is it `from plugins import api` or `import framework.plugins.api` etc? Whatever you do it should be the module import names you use instead of filenames (i.e. no full paths, /s or `.py` suffixes). — pullmyteeth, Dec 14 '20 at 18:05

score 2 · Answer 1 · edited May 23 '17 at 11:45

I've had the problem alike packing PyQt application with Py2Exe (I've also struggled with PyInstaller and cx_freeze, but only py2exe helped me).

Here's the detailed solution. I've added them explicitly as:

data_files += [('source', glob('source/*.py'),)]
setup(
      data_files=data_files,
      ....  # other options
      windows=[
        {
          "script": "launcher.py",
          "icon_resources": [(0, "resources/favicon.ico")]
        }
     )

Then I import them and call. Hope this approach will be useful.

How to add dynamic python modules to PyInstaller's specs?

1 Answers1