Split Python source into separate directories?

Question

Here are some various Python packages my company "foo.com" uses:

com.foo.bar.web
com.foo.bar.lib
com.foo.zig.web
com.foo.zig.lib
com.foo.zig.lib.lib1
com.foo.zig.lib.lib2

Here's the traditional way to store the source on disk:

pysrc/
  com/
    foo/
      bar/
        web/
        lib/
      zig/
        web/
        lib/
          lib1/
          lib2/

PYTHONPATH=pysrc

But for organizational purposes (different teams, different revision control, etc.), we want to store these as follows:

bar/
  pysrc/
    com/
      foo/
        bar/
          web/
          lib/
zig/
  pysrc/
    com/
      foo/
        zig/
          web/
          lib/
            lib1/
            lib2/

PYTHONPATH=bar/pysrc:zig/pysrc

The question is:

Are there any issues with this second method of organization?

For example, if we import com.foo, where would Python look for the __init__.py?

Would symlinking these directories make sense? e.g.:

pysrc/
  com/
    foo/
      bar/ -> symlink to /bar/pysrc/com/foo/
      zig/ -> symlink to /zig/pysrc/com/foo/

Any general code organizational suggestions are welcome.

possible duplicate of [Python accessing modules from package that is distributed over different directories](http://stackoverflow.com/questions/2847146/python-accessing-modules-from-package-that-is-distributed-over-different-director) — Ignacio Vazquez-Abrams, Jun 18 '10 at 19:52
@Ignacio: Thanks, I didn't see that one. I've added here the question about symlinking, which wasn't discussed in the other one, and seems like a viable solution. — dkamins, Jun 18 '10 at 19:59
This works in Python 3.3+, see http://stackoverflow.com/a/17202582/851737 — schlamar, May 28 '14 at 12:53

score 4 · Accepted Answer · answered Jun 18 '10 at 20:00

Python will go through sys.path in order (which includes PYTHONPATH and then some), looking for a com.foo package in each. The first one it finds, it will use to the exclusion of others, unlike Perl or Java which effectively merges together package namespaces. There are things you can do to __path__ that change this behavior, but "first match wins" is how Python behaves out of the box.

As long as you keep all of com.foo.bar entirely in bar/ and all of com.foo.zig entirely in zig/, you shouldn't have any problems with the second layout.

score 3 · Answer 2 · answered Feb 16 '18 at 19:23

Reading through the PEP 420 page, it looks like you can add the following __init__.py to the shared packages:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

So your directory structure would be as follows (*-marked __init__.py files have the above code):

myroot/
├── bar
│   └── pysrc
│       └── com
│           ├── ****__init__.py****
│           └── foo
│               ├── ****__init__.py****
│               └── bar
│                   ├── __init__.py
│                   ├── lib
│                   │   ├── __init__.py
│                   │   └── barlib.py
│                   └── web
│                       ├── __init__.py
│                       ├── barweb.py
└── zig
    └── pysrc
        └── com
            ├── ****__init__.py****
            └── foo
                ├── ****__init__.py****
                └── zig
                    ├── __init__.py
                    ├── lib
                    │   ├── __init__.py
                    │   ├── lib1
                    │   │   ├── __init__.py
                    │   │   └── ziblib1.py
                    │   └── lib2
                    │       ├── __init__.py
                    │       └── ziblib2.py
                    └── web
                        ├── __init__.py
                        ├── zigweb.py

Set the python path to point to your com/ directories:

barPath=/myroot/bar/pysrc/
zigPath=/myroot/zig/pysrc/
export PYTHONPATH=$PYTHONPATH:$barPath:$zigPath

To test (I tried on 2.7.14 and 3.6.4):

from com.foo.bar.web.barweb import BarWeb
from com.foo.zig.web.zigweb import ZigWeb
b = BarWeb()
z = ZigWeb()

Not having the __init__.py code yields:

ImportError: No module named zig.web.zigweb

Split Python source into separate directories?

2 Answers2

Linked