Restructuring a directory of python scripts for easier use in another package

Question

I have a non-python model with several python scripts in scripts sub-directory. If we use the model in a project, we usually clone it into a model sub-directory of the project. This works, but it means that the python scripts have to be called as

python model/scripts/do_something.py <ARGS>

Also note that the python scripts expect to find the model at ../, relative to their location.

I want to simplify the usage by making a pip-installable python package that would install one binary (entry point) called hyopt, so all the common actions could be done as hyopt <ACTION> <ARGS>.

To be able to access the python scripts from the new wrapper, I have landed on the following structure of the python project:

│   pyproject.toml
│   README.md
└───src
    └───hyopt
        │   hyopt_cli.py
        ├───model
        │   │   <model files>
        │   └───scripts
        │       │   <python scripts>

There, hyopt_cli.py is the new wrapper and model directory includes the existing code as a git submodule. This structure allows me to run the existing scripts as a subprocess, using importlib.resources to locate them from hyopt_cli.py.

However, I would prefer to access some functions and methods in the python files directly from hyopt_cli.py and there I encountered the following problem: Let's say I have three python files there, including (between other) the following:

# foo.py

class Foo:
    ...

# bar.py

from foo import Foo

def run_model(args):
    ...

# runner.py

from bar import run_model
import argparse

parser = argparse.ArgumentParser()
...
args = parser.parse_args()
run_model(args)

Here, I can run runner.py from hyopt_cli.py using subprocess.run(), but I would prefer to call run_model() directly. However, doing

from .model.scripts.bar import run_model

fails with No module named 'foo'.

I am still new to python packaging, but I believe this is because the scripts directory is not a package in the python sense, but rather a collection of python files (mostly scripts, with some files defining common classes etc). It does not have __init__.py, either.

Based on this SO answer, I found that I can make it work by adding the following to hyopt_cli.py, before any import from the scripts directory:

import sys
from importlib import resources

script_dir = (resources.files('hyopt.model.scripts') / 'runner.py').parent
sys.path.append(str(script_dir))

The " / 'runner.py').parent" part could be skipped if I added __init__.py to the scripts folder (without it, resources.files() returns MultiplexedPath which cannot be converted to string).

While this works, it feels more like a hack than a solution. Is there a more pythonic way to make the importing from hyopt_cli.py work, without affecting the current functionality of the existing model package? In other words, it should still be possible to use the existing repo without the new python wrapper and call the existing scripts with the same syntax as before (as shown above).

Actually, I am willing to drop the 'same syntax' requirement, if it makes the solution easier/clearner/more pythonic.

Restructuring a directory of python scripts for easier use in another package

0 Answers0