0

I have a package that sort of looks like this:

- package
    -- module1.py
    -- module2.py
    -- __init__.py

In init.py I am programmatically scanning the package and importing the modules that are inside.

import importlib
import pkgutil

registry = {}


def creatable(cls):
    if cls.handles_type() in registry:
        raise KeyError("Duplicate string representations found for string: " + cls.handles_type())

    registry[cls.handles_type()] = cls

    return cls


def import_submodules(package, recursive=False):
    """ Import all submodules of a module, recursively, including subpackages
    """
    if isinstance(package, str):
        package = importlib.import_module(package)

    results = {}

    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = package.__name__ + '.' + name
        results[full_name] = importlib.import_module(full_name)

        if recursive and is_pkg:
            results.update(import_submodules(full_name))

    return results


import_submodules(__name__)

Inside the modules, there are classes annotated with the @creatable decorator that I define here. The idea is to have a registry dictionary with the key being the class name and the value being the class (so I can create instances using the string representation).

The issue is, the registry is empty when using Pyinstaller.

UPDATE:

The issue is that package.__path__ does not contain anything. It can be fixed by adding the .py files on the package path i.e.

 datas=[
    ('package/*.py', 'package'),
 ]

But this doesn't look like a good solution - i.e. I'd be sending code to the end user.

Cœur
  • 37,241
  • 25
  • 195
  • 267
Hydroxis
  • 95
  • 1
  • 12
  • Read the documentation about hidden imports https://pyinstaller.readthedocs.io/en/stable/when-things-go-wrong.html#listing-hidden-imports – DisappointedByUnaccountableMod Dec 04 '19 at 08:49
  • 1
    @barny hiddenimports would work if I explicitely tell which modules are hiddenly imported. In my case, I am importing all the modules under a package without knowing anything about them. They need to be somehow dynamically resolved. – Hydroxis Dec 04 '19 at 09:28
  • FYI PyInstaller always sends your code to the end user, albeit slightly obfuscated in pyc - but don't imagine/guess/hope this isn't decompilable, because it is. The source of your problem is nothing to do with creating your registry and everything to do with your design of importing modules automatically - presumably wanting to avoid updating __init__.py as you add new modules - ref https://stackoverflow.com/questions/1057431/how-to-load-all-modules-in-a-folder – DisappointedByUnaccountableMod Dec 05 '19 at 11:31
  • You could try two things, neither of which I have tried: 1. use the `datas` but use a folder which only has the pyc files, or 2. write yourself an analysis hook which at pyinstaller run time lists all the .py files in your package as hidden imports - see https://github.com/pyinstaller/pyinstaller/issues/2572 and https://gist.github.com/ddcatgg/178aeee927c4bfb0e7e38071c95cbcce - this is really the nicest solution because then the pyc files get embedded in the bootloader binary in your bundle whether one-file or one-folder. – DisappointedByUnaccountableMod Dec 05 '19 at 11:32
  • @barny - I know, the issue is mostly that the solution is not 'idiomatic' (i.e. datas is there to transfer actual data, not python modules), and you're right about updating init. I will try #2 and report back. – Hydroxis Dec 05 '19 at 12:13
  • No not updating init - if you use a pyinstaller hook you can dynamically specify the hidden imports - so these will be available alongside the init.py without having to modify it. I had a look and what seems to happen is that local packages/modules are embedded in an internal directory structure in the bootloader, and tried it with a hardcoded `hiddenimports[]` generated by a hook, so it should work with a more sophisticated hook that specifies *.py (except __init__.py) that it finds in the package folder. And then you add e.g. --additional-hooks-dir hooks into your pyinstaller command. – DisappointedByUnaccountableMod Dec 05 '19 at 14:49
  • @barny I didn't have a chance to try it out, but I don't think It's going to work because of the way the import is done (by searching package.__path__) which seems to be empty. – Hydroxis Dec 05 '19 at 14:52
  • Feels close, though. It might be that you don't need the dynamic importing but have to add an explicit import of the package and `from package import *` and then normal python loading does the rest. – DisappointedByUnaccountableMod Dec 05 '19 at 15:35
  • 1
    So it looks like there's a way using first the hook to scan the package folder to provide the hidden imports - this ensures they are packaged in the bundle - and then second in the init.py detect that this is frozen and deduce the module names from the __loader__.toc (scan for package.* as provided in the hidden import list) and import, or if not frozen then import the current way. Oh and you'll need an explicit import of package. – DisappointedByUnaccountableMod Dec 05 '19 at 16:55
  • @barny the import works, but the creatable annotation logic does not. – Hydroxis Dec 06 '19 at 11:58

1 Answers1

1

This isn't an answer but is the only way I can show some code - this is my hook file for package mypkg, in the hooks folder, called hook-mypkg.py

import os

imports = []

for root, dirs, files in os.walk(os.path.join(os.getcwd(),"mypkg" )):
    print(root)
    for file in files:
        if file.endswith( ".py") and not file.endswith( "__init__.py"):
            print( "  ",file)
            imports.append("mypkg."+file[:-3])
    print( "........." )
print( "hi=",imports )

hiddenimports = imports

This definitely works to include the .py files it finds in the bundle - they appear in globals()['__loader__'].toc in init.py as e.g. "mypkg.file1" for module mypkg\file1.py when sys.frozen is True.

Note after editing the hook you have to delete the dist and build folders then re-run pyinstaller

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22