1

I have been trying to use Hydra with PyInstaller and failed. I have created a very configuration example similar to example in here.

I noticed that hydra packages are not being found by PyInstaller so I created a simple hook file hook-hydra.py with the following code:

from PyInstaller.utils.hooks import collect_data_files, collect_submodules

datas = collect_data_files('hydra')
hiddenimports = collect_submodules('hydra')

That seemed to solve the module imports failures, but then when I tried to run the executable in the command line I got the following error:

Traceback (most recent call last):
  File "lib\site-packages\hydra\_internal\utils.py", line 198, in run_and_report
  File "lib\site-packages\hydra\_internal\utils.py", line 321, in <lambda>
  File "lib\site-packages\hydra\_internal\hydra.py", line 74, in create_main_hydra2
  File "lib\site-packages\hydra\_internal\config_loader_impl.py", line 80, in __init__
  File "lib\site-packages\hydra\_internal\config_repository.py", line 22, in __init__
  File "lib\site-packages\hydra\_internal\sources_registry.py", line 30, in resolve
ValueError: No config source registered for schema pkg, supported types : []

I can't seem to figure it out, any ideas?
I'm using PyInstaller 3.6 and Hydra 1.0.4

Yossi
  • 69
  • 2
  • 7

4 Answers4

2

After taking a look at PyInstaller, it looks like it's attempting to discover needed packages and it somehow fails to do a good job for Hydra. Hydra has some built in plugins that are discovered at runtime, including the config sources. The error suggests that the config sources were not packages by PyInstaller.

If PyInstaller is trying to be clever and only include things it sees a direct dependency on it is likely to fail for Hydra. Try to add all hydra modules explicitly to your PyInstaller config file.

As an alternative packaging method for your app, take a look at the application packaging example here. It shows you the supported method for installing Hydra apps (with a working example).

Omry Yadan
  • 31,280
  • 18
  • 64
  • 87
  • the hook file that I created suppose to add all the sub modules under Hydra, is config sources a module? is it located under the hydra folder? – Yossi Dec 01 '20 at 13:40
  • No. config soruces are here:https://github.com/facebookresearch/hydra/tree/master/hydra/_internal/core_plugins – Omry Yadan Dec 01 '20 at 17:20
  • tried using the global hidden-imports and to include all sub modules and I'm still getting the same error. ill try using OmegaConf and handled it in my code. thanks. – Yossi Dec 02 '20 at 15:25
1

Hydra does not play nice with PyInstaller because it dynamically reads the plugin packages using pkgutil.walk_packages.

Check out the issue and answers here: Pyinstaller - include programmatically imported modules

As you are not solving this from a library developer's point of view, copying the package using the data field in the .spec file could solve your issue, although it is a bit hacky. Either this or switching to OmegaConf is a good solution in my opinion.

fblthp
  • 98
  • 5
  • 1
    I tried different things and couldnt get it to work. I ended up using just the OmegaConf package and created my own wrappers around it and it worked perfectly. Thanks for your answer. – Yossi Mar 22 '21 at 11:02
0

I faced similar issues and ended up having to create a pyinstaller hook named hook-hydra.py and put it in the pyinstaller hooks folder. It could probably be reduced to lower level hydra namespaces. Leveraging other hooks, I did the following:

Only issue I face is inline using compose overrides=[f"environment.root_path={set_root_path}"], complains about EOF .... etc. But have workaround for that. Logging and Fetching configuration works like a charm.

from PyInstaller.utils.hooks import collect_submodules

#  Use this to get hydra internals
#  "hydra._internal.core_plugins"
# "hydra.plugins"
# 'hydra._internal.core_plugins','hydra.grammar.gen.OverrideParser'

hiddenimports = collect_submodules('hydra')
DinoDon
  • 41
  • 4
  • I have one more piece I will share here shortly, I forget I also have some hook logic to include the hydra\conf default yaml files; Else you get the In 'hydra/config': Could not find 'hydra/env/default' error. – DinoDon Feb 04 '22 at 15:14
  • Pyinstaller provides the `--runtime-hook` and `--additional-hooks-dir` options (or their spec file equivalents) that can be used instead of adding files into the library's hooks dir directly – Numerlor Feb 04 '22 at 18:55
  • Agree, better path\strategy. I had sporadic success with --additional-hooks-dir option in spec file so I went with mentioned above. – DinoDon Feb 04 '22 at 20:29
0

This works for me, no problems running from command line, using PyInstaller 4.5.1, Hydra 1.2:

from PyInstaller.utils.hooks import collect_data_files, collect_submodules

datas = collect_data_files('hydra', subdir='conf')
hiddenimports = collect_submodules('hydra')
rlzzz
  • 1
  • 1