Python folder structure for project directory and easy import

Question

My team has a folder of several small projects in python3. Amongst them, we have a utility folder with several utility functions, that are used throughout the projects. But the way to import it is very uncomfortable. This is the structure we use:

temp_projects
    util
        storage.py
        geometry.py
    project1
        project1.py
    project2
        project2.py

The problem is that the import in the projects looks terrible:

sys.path.insert(1, os.path.join(sys.path[0], '..'))
import util.geometry

util.geometry.rotate_coordinates(....)

Also, pycharm and other tools are having trouble understanding it and to supply completion.

Is there some neater way to do that?

Edit: All of the projects and utils are very much work-in-progress and are modified often, so I'm looking for something as flexible and comfortable as possible

Use setuptools. https://setuptools.readthedocs.io/en/latest/setuptools.html — Stephen Rauch, Sep 23 '18 at 17:34
I think that the following thread answers your question: https://stackoverflow.com/questions/19917492/how-to-use-pythonpath. Is there any reason why this thread does not help you? — tangoal, Sep 27 '18 at 19:44

score 2 · Accepted Answer · answered Sep 26 '18 at 21:30

2

PYTHONPATH environment variable might be a way to go. Just set it to projects folder location:

PYTHONPATH=/somepath/temp_projects

and you'll be able to use util as follows:

import util.geometry

util.geometry.rotate_coordinates(....)

Also this will be recognized by PyCharm automatically.

answered Sep 26 '18 at 21:30

Vader

3,675
23
40

I tried several options and I came to the same result than you. By the way, maybe you should not overwrite the PYTHONPATH, but just add the project path to the PYTHONPATH. – tangoal Sep 27 '18 at 19:43

score 1 · Answer 2 · answered Sep 23 '18 at 17:06

I believe the correct route would be completely different than what you are doing right now. Each project should be stored in a different Git repository, share modules should be added as git submodules. Once those projects will get larger and more complex (and they probably will), it would be easier to manage them separately.

In short

Projects structure should be:

Project_1
  |- utils <submodule>
       |- storage.py
       |- geometry.py
  |- main.py

Project_2
  |- utils <submodule>
       |- storage.py
       |- geometry.py
  |- main.py

Working with submodules

### Adding a submodule to an existing git directory
git submodule add <git@github ...> <optional path/to/submodule>

### Pulling latest version from master branch for all submodules
git submodule update --recursive --remote

### Removing submodule from project
# Remove the submodule entry from .git/config
git submodule deinit -f path/to/submodule

# Remove the submodule directory from the project's .git/modules directory
rm -rf .git/modules/path/to/submodule

# Remove the entry in .gitmodules and remove the submodule directory located at path/to/submodule
git rm -f path/to/submodule

Further reading https://git-scm.com/book/en/v2/Git-Tools-Submodules

The question is about project structure, one possible solution is using git. — Michael liv., Sep 28 '18 at 20:38
I don't think that it is a good idea to depend on the version control. How do you assure that the scripts run on the target machine after installing them? Git is not available on the target. — tangoal, Oct 04 '18 at 10:48

score 1 · Answer 3 · answered Sep 28 '18 at 08:03

According to Importing files from different folder adding a __init__.py in the util folder will cause python to treat it as a package. Another thing you can do is use import util.geometry as geometry and then you could use geometry.rotate_coordinates(...) which also improves readability.

score 0 · Answer 4 · answered Sep 27 '18 at 11:13

If you create a setup.py file for your util module you can install it simply with pip. It will handle everything for you. After installation you can import it across the system.

import util

pip install

# setup.py is in current folder
sudo -H pip3 install .

or if the util module itself is still under development you can install it with the -e editable option. Then it automatically updates the installation when you make changes to the code.

sudo -H pip3 install -e .

For the project administration I recommend to use git as @Michael liv. is suggesting, especially if you work in a team.

score -1 · Answer 5 · answered Sep 26 '18 at 14:13

Use importlib.

import importlib, importlib.util

def module_from_file(module_name, file_path):
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

geometry = module_from_file("geometry", "../temp_projects/utils/geometry.py")

geometry.rotate_coordinates(...)

score -1 · Answer 6 · answered Sep 27 '18 at 09:03

Other option (I used this appoach in my project):

Lets assume that projects are ran from project1.py and project2.py files accordingly.

At the top of these files you can add following imports and actions:

import sys
import os

sys.path.append(os.path.join(os.getcwd(), os.pardir))

import your_other_modules

your_other_modules.py will contains following for impotring utils

from utils import storage
from utils import geometry
# or from project2 import project2, etc..

May be it's not a best way, but just like one more option. I hope it will be helpful for someone.

Python folder structure for project directory and easy import

6 Answers6

Linked