PyInstaller ModuleNotFoundError --paths flag seems to not work

Question

I build a simple GUI using Tkinter which I would like to freeze to a standalone executable. I am doing this inside a conda environment. Working with OSX 10.15.7, python 3.7, PyInstaller 4.5.1 and conda 4.10.0. The folder structure looks like this (simplified):

 - ImSep_files
  | - ai4eutils
  | - ImSep
  |  | - ImSep_GUI.py
  | - cameratraps
     | - detection
        | - run_tf_detector.py

The script calls other scripts in the ai4eutils and cameratraps folders. If I create a conda env, set the PYTHONPATH to include the paths to ai4eutils and cameratraps, and run python ImSep_GUI.py, there is no problem. The GUI opens and functions perfectly. However, if I do exactly the same but run pyinstaller instead of python, it creates an exe which opens the GUI but throws an error when a button is pressed.

  File "/Users/peter/Applications/ImSep_files/cameratraps/detection/run_tf_detector_batch.py", line 56, in <module>
    from detection.run_tf_detector import ImagePathUtils, TFDetector
ModuleNotFoundError: No module named 'detection.run_tf_detector'

This means that pyinstaller cannot find the run_tf_detector.py file. I have tried adding the --paths flag like:

pyinstaller --onefile --windowed --name='ImSep' --icon='imgs/logo_small_bg.icns' --paths=/Users/peter/Applications/ImSep_files --paths=/Users/peter/Applications/ImSep_files/ai4eutils --paths=/Users/peter/Applications/ImSep_files/cameratraps --paths=/Users/peter/Applications/ImSep_files/cameratraps/detection ImSep_GUI.py

I am aware that there are many topics about this type or error. I have tried many potential solutions, but none seem to work. I have tried the following:

• Using the --hidden-import flag, as suggested here by HHest. If tried different versions: --hidden-import detection.run_tf_detector, --hidden-import cameratraps.detection.run_tf_detector, --hidden-import cameratraps.detection, etc.
• Adjusting the hiddenimports=[], line with the above paths, as suggested here by user1251007.
• Adding sys.path.append(path/to/run_tf_detector.py) to the top of ImSep_GUI.py.
• Downgrading pyinstaller to 3.1, as suggested here by fivef.
• Creating a hook.py with detection.run_tf_detector the in a hooks folder and adding it as --additional-hooks-dir=hooks, as suggested here by Legorooj.
• Loading the needed module as data in the spec file, as suggested here by Ken4scholars.
• Copy run_tf_detector.py into the file folder the same level of ImSep.exe, as suggested here by Wayne Zhang.
• Call pyinstaller from parent directory, as suggested here by all or None.
• Install pyinstaller in the same directory in which ImSep_GUI.py is present, as suggested here by Habeeb Rahman K T.
• Install pyinstaller using conda-forge instead of pip, as suggested here by piping piping.

FYI, this is how I create the environment and run pyinstaller:

conda create --name imsepcondaenv python=3.7 -y
conda activate imsepcondaenv
pip install tensorflow==1.14 pillow==8.4.0 humanfriendly==10.0 matplotlib==3.4.3 tqdm==4.62.3 jsonpickle==2.0.0 statistics==1.0.3.5 requests==2.26.0
conda install -c conda-forge pyinstaller -y
cd ~/Applications/ImSep_files
export PYTHONPATH="$PYTHONPATH:$PWD/ai4eutils:$PWD/cameratraps"
cd ImSep
pyinstaller --onefile --windowed --name='ImSep' --icon='imgs/logo_small_bg.icns' --paths=/Users/peter/Applications/ImSep_files --paths=/Users/peter/Applications/ImSep_files/ai4eutils --paths=/Users/peter/Applications/ImSep_files/cameratraps --paths=/Users/peter/Applications/ImSep_files/cameratraps/detection ImSep_GUI.py

Does anyone have an idea of what I'm doing wrong?

PS: For OSX and UNIX users it is possible to get a reproducible example:

mkdir ImSep_files
cd ImSep_files
git clone https://github.com/Microsoft/cameratraps -b tf1-compat
git clone https://github.com/Microsoft/ai4eutils
git clone https://github.com/PetervanLunteren/ImSep.git
curl --output md_v4.1.0.pb https://lilablobssc.blob.core.windows.net/models/camera_traps/megadetector/md_v4.1.0/md_v4.1.0.pb

Answer 1

PYTHONPATH is almost always a local minimum. In my experience, it only complicates things in the long run. I would recommend Step 1 is remove PYTHONPATH from your workflow and learn about python packagens and editable intsalls. It'll make development much easier in the long run.

PYTHONPATH basically started as a way to let "scripts" access other modules without actually installing a package. This made more sense back in the bad old days before virtualenv and conda, but now it is just easier and more organized to just use a package structure.

Try structuring your project like a typical installable python library. E.g.

.
├── .git
├── ImSep_files
│  ├── ai4eutils
│  ├── cameratraps
│  │  └── detection
│  │     └── run_tf_detector.py
│  └── ImSep
│     └── ImSep_GUI.py
└── setup.py

Make sure you can pip install . from your root directory. You should have some top-level package name you import from (in this case I've arbitrarily picked ImgSep_Files as your library name, but it could be whatever). Then you ought to be able to always import using either absolute or relative package syntax, like

from .detection.run_tf_detector import ImagePathUtils, TFDetector

The ultimate test is if you can run python -m ImSep_files.cameratraps.detection.run_tf_detector. Without using PYTHONPATH. That means you have your import structured correctly and pyinstaller should have no problem picking up on your dependencies.

Update: here's an example simple package with setup.py. I chose setup.py even though that's kinda old school and things are moving towards pyproject.toml because there is more documentation out there for this style:

from setuptools import setup, find_packages

setup(
    name="my_package",
    description="An example setup.py",
    license="MIT",
    packages=find_packages(),
    python_requires=">=3.7",
    zip_safe=False,
    install_requires=[
        "tensorflow",
    ],
    classifiers=[
        "Programming Language :: Python :: 3.7",
    ],
    entry_points={
        "console_scripts": [
            "run_tf_detector=my_package.scripts.run_tf_detector:main",
            "imsep_gui=my_package.gui.gui:main",
        ]
    },
)

Then I have a layout like this:

.
└── my_project_name
   ├── .git
   ├── my_package
   │  ├── gui
   │  │  ├── gui.py
   │  │  └── gui_utils.py
   │  ├── scripts
   │  │  └── run_tf_detector.py
   │  └── detection
   │     └── tf_detector.py
   ├── README.md
   ├── setup.py
   └── tests
      └── test_tf_detector.py

my_project_name is my "repo root". my_package is the name of my package. I would import like from my_package.detection.tf_detector import TFDetector. In this case, I would put all of the classes and logic in tf_detector.py, and then run_tf_detector.py is basically just:

import sys
from my_package.detection.tf_detector import TFDetector


def main(args=None):
    args = args or sys.argv
    detector = TFDetector()
    detector.detect(args)

if __name__ == __main__:
    main()

The GUI follows a simple pattern, with gui.py containing the entry point to start the gui. This kind of organization keeps your functional code separate from the nuts and bolts of running as a script. It makes it easy for example to have detectors which run as a CLI script, or as part of a GUI, or as a library you can import.

Entry points are used to tell the installer "this is a thing that you run or a plugin". Some more info.