0

I started to build a data processing framework in Python 3. The goal is to develop own packages and write example code to use the packages. The framework will be shared (any maybe developed) with other researchers by git.

What is the best way to handle the search paths on multiple computers and how should the __init__.py files look like in the directories?

I was thinking about adding sys.path relative to the main_directory (Python: Best way to add to sys.path relative to the current running script) with sys.path.append(os.getcwd()) or sys.path.append(".."). I don’t know if this is a good way to import beyond top level on different machines with different absolute paths and operation systems. (The beyond top level package error/problem: beyond top level package error in relative import)

The basic directory structure might be (I will add more packages and processing functions later):

main_directory/
main_directory/__init__.py
main_directory/packages/
main_directory/packages/__init__.py
main_directory/packages/preprocessing/
main_directory/packages/preprocessing/filters.py    #implementation of multiple classes: e.g. fillter1 and filter2
main_directory/packages/preprocessing/__init__.py
main_directory/packages/postprocessing/
main_directory/packages/postprocessing/filters.py   #implementation of multiple classes: e.g. fillterA and filterB
main_directory/packages/postprocessing/__init__.py
main_directory/examples/
main_directory/examples/easy_example.py #file with examples to use the filters
main_directory/your_code/
main_directory/your_code/your_code.py   #file with your code to use the filters
walfab.dev
  • 3
  • 3

1 Answers1

1

There is a standard package layout. If you follow that you can deploy your programs without having to touch sys.path.

Use the resources on this website and learn setuptools.

https://packaging.python.org/tutorials/packaging-projects/

Update

During development you can use a virtual environment. First create one:

$ virtualenv venv

Installing your package with -e creates a link to your directory.

$ venv/bin/pip install -e yourproject.git

When you start the python in your virtual environment your imports should work.

$ venv/bin/python
>>> import preprocessing.filters
>>>
Eddy Pronk
  • 6,527
  • 5
  • 33
  • 57
  • Thank you @Eddy Pronk. This is a very elegant solution for the “final” framework. Although, I think this solution is not very practical in a developing and debugging phase, because I have to build and install the package all the time. Or, do I understand it wrongly? Would you recommend developing and debugging the framework by adding the framework path to my _sys.path_? – walfab.dev Jul 27 '18 at 12:16
  • @walfab.dev During development you can use a virtual environment. See update. – Eddy Pronk Jul 27 '18 at 12:23