159

In Python, a namespace package allows you to spread Python code among several projects. This is useful when you want to release related libraries as separate downloads. For example, with the directories Package-1 and Package-2 in PYTHONPATH,

Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

the end-user can import namespace.module1 and import namespace.module2.

What's the best way to define a namespace package so more than one Python product can define modules in that namespace?

joeforker
  • 40,459
  • 37
  • 151
  • 246
  • 8
    It looks to me like module1 and module2 are actually subpackages rather than modules. As I understand it, a module is basically a single file. Maybe subpkg1 and subpkg2 would make more sense as names? – Alan Dec 24 '14 at 15:44

5 Answers5

104

TL;DR:

On Python 3.3 you don't have to do anything, just don't put any __init__.py in your namespace package directories and it will just work. On pre-3.3, choose the pkgutil.extend_path() solution over the pkg_resources.declare_namespace() one, because it's future-proof and already compatible with implicit namespace packages.

Python 3.3 introduces implicit namespace packages, see PEP 420.

This means there are now three types of object that can be created by an import foo:

  • A module represented by a foo.py file
  • A regular package, represented by a directory foo containing an __init__.py file
  • A namespace package, represented by one or more directories foo without any __init__.py files

Packages are modules too, but here I mean "non-package module" when I say "module".

First it scans sys.path for a module or regular package. If it succeeds, it stops searching and creates and initalizes the module or package. If it found no module or regular package, but it found at least one directory, it creates and initializes a namespace package.

Modules and regular packages have __file__ set to the .py file they were created from. Regular and namespace packages have __path__set to the directory or directories they were created from.

When you do import foo.bar, the above search happens first for foo, then if a package was found, the search for bar is done with foo.__path__as the search path instead of sys.path. If foo.bar is found, foo and foo.bar are created and initialized.

So how do regular packages and namespace packages mix? Normally they don't, but the old pkgutil explicit namespace package method has been extended to include implicit namespace packages.

If you have an existing regular package that has an __init__.py like this:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

... the legacy behavior is to add any other regular packages on the searched path to its __path__. But in Python 3.3, it also adds namespace packages.

So you can have the following directory structure:

├── path1
│   └── package
│       ├── __init__.py
│       └── foo.py
├── path2
│   └── package
│       └── bar.py
└── path3
    └── package
        ├── __init__.py
        └── baz.py

... and as long as the two __init__.py have the extend_path lines (and path1, path2 and path3 are in your sys.path) import package.foo, import package.bar and import package.baz will all work.

pkg_resources.declare_namespace(__name__) has not been updated to include implicit namespace packages.

clacke
  • 7,688
  • 6
  • 46
  • 48
  • 4
    What about setuptools? Do I have to use the `namespace_packages` option? And the [`__import__('pkg_resources').declare_namespace(__name__)`](http://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages) thing? – kawing-chiu Oct 10 '16 at 01:36
  • 6
    Should I add `namespace_packages=['package']` in the `setup.py`? – Laurent LAPORTE Oct 10 '16 at 08:25
  • @laurentlaport Good question! If you are on Python 3.3+, I hope not, but please experiment and come back with an answer. ;-) I looked at http://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages , but it's not clear if this is legacy documentation that has not been updated, or if it's still valid. – clacke Oct 10 '16 at 09:31
  • @kawingchiu Please see my comment to `laurentlaport`. `declare_namespace` is obsolete, from what I understand, but the `namespace_packages` option needs investigation. (NOTE: I base this on what I have read. I have not released any packages using Setuptools.) – clacke Oct 10 '16 at 09:34
  • Yes to both kawing-chiu and laurentlaport. Tested on Python 3.4.4 – villasv Nov 29 '16 at 17:57
  • 2
    @clacke: With `namespace_packages=['package']`, setup.py will add a ``namespace_packages.txt`` in the EGG-INFO. Still don't know the impacts… – Laurent LAPORTE Dec 09 '16 at 11:55
  • Good to know, whatever it means. :-) Hope someone else (or you, when you figured it out) can fill in the details. – clacke Dec 09 '16 at 13:59
  • 1
    @kawing-chiu The benefit of `pkg_resources.declare_namespace` over `pkgutil.extend_path` is that it will continue to monitor `sys.path`. That way, if a new item is added to `sys.path` after a package in the namespace is first loaded then packages in the namespace in that new path item are still able to be loaded. (A benefit of using `__import__('pkg_resources')` over `import pkg_resources` is that you don't end up `pkg_resources` being exposed as `my_namespace_pkg.pkg_resources`.) – Arthur Tacca Jan 06 '17 at 10:31
  • @ArthurTacca Interesting. How does it do that? It adds module resolution code to the package instead of just extending `__path__`? – clacke Jan 12 '17 at 02:32
  • 1
    @clacke It doesn't work that way (but it has the same effect as if it did). It maintains a global list of all package namespaces created with that function, and watches `sys.path`. When `sys.path` changes it checks if that affects the `__path__` of any namespace, and if it does then it updates those `__path__` properties. – Arthur Tacca Jan 12 '17 at 09:03
  • 1
    Actually, I'm not sure how it does it. The source code is here: https://github.com/pypa/setuptools/blob/7c0c39ef1f60571709e5a8e6680f2147c38cd4ff/pkg_resources/__init__.py – Arthur Tacca Jan 12 '17 at 09:14
81

There's a standard module, called pkgutil, with which you can 'append' modules to a given namespace.

With the directory structure you've provided:

Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

You should put those two lines in both Package-1/namespace/__init__.py and Package-2/namespace/__init__.py (*):

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

(* since -unless you state a dependency between them- you don't know which of them will be recognized first - see PEP 420 for more information)

As the documentation says:

This will add to the package's __path__ all subdirectories of directories on sys.path named after the package.

From now on, you should be able to distribute those two packages independently.

Luis Masuelli
  • 12,079
  • 10
  • 49
  • 87
Mike Hordecki
  • 92,113
  • 3
  • 26
  • 27
  • 19
    What are the pros and cons of using that versus __import__('pkg_resources').declare_namespace(__name__) ? – joeforker Nov 04 '09 at 20:11
  • 15
    First, `__import__` is considered bad style in this case as it can be easily replaced with plain import statement. More to the point, pkg_resources is a non-standard library. It comes with setuptools, so that's a non-issue, though. Quick googling reveals that pkgutil was introduced in 2.5 and pkg_resources predates it. Nevertheless, pkgutil is a oficially-recognized solution. pkg_resources inclusion was, in fact, rejected in PEP 365. – Mike Hordecki Nov 04 '09 at 21:17
  • Their functionality is probably the same, though, as zope.interface uses both in it's `__init__.py`. – Mike Hordecki Nov 04 '09 at 21:19
  • @MikeHordecki pkgutil was introduced in python 2.3 actually. – Jens Timmerman Mar 30 '12 at 09:21
  • Rejection to be included as part of the standard library doesn't necessarily imply outright rejection of the idea as unsound. Maybe it was rejected specifically so that people could continue to use the feature in setuptools. The only difference I can discern is that one requires you to be using setuptools (and hence will work with zipped eggs, etc), while the other does not. – Jonathan Hartley Jul 10 '12 at 09:26
  • 3
    Quote from [PEP 382](http://www.python.org/dev/peps/pep-0382/): The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports *.pkg files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't. – Drake Guan Sep 26 '12 at 02:24
  • 8
    Shouldnt these two lines be put into both files: `Package-1/namespace/__init__.py` *and* `Package-2/namespace/__init__.py` providing that we don't know which Package dir is listed first? – Bula Apr 11 '13 at 13:55
  • Wondering the same of @Bula – alanjds Jul 11 '13 at 23:17
  • 1
    @Bula: My own testing of this shows that it's only needed for the package that is listed first. But if you don't know, put them in both. At least I haven't had any troubles with doing it that way. – Christoffer Karlsson Aug 02 '13 at 05:48
  • 3
    @ChristofferKarlsson yes that is the point, it is all right if you know which is first, but the real question is can you guarantee that it will be the first in any situation, i.e. for other users? – Bula Aug 06 '13 at 08:23
  • 3
    @Bula: You are correct. From [PEP 420](http://www.python.org/dev/peps/pep-0420/): "Every distribution needs to provide the same contents in its `__init__.py`, so that `extend_path` is invoked independent of which portion of the package gets imported first." – Søren Løvborg Jan 08 '14 at 14:07
  • @MikeHordecki: Can you take a look at my new question http://stackoverflow.com/questions/27545177/namespace-package-with-both-tarball-and-egg-in-python – nam Dec 18 '14 at 11:16
  • As another voice for `pkgutil` over `pkg_resources`: I've found that `pkg_resources` doesn't correctly respect `PYTHONPATH` order when you have two `__init__.py` files that both contain the `declare_namespace` call, but `pkgutil` does. – Haldean Brown Mar 29 '16 at 20:52
  • @joeforker on why `import__('pkg_resources').declare_namespace(__name)` is discouraged, [this might be a good read](https://github.com/ncoghlan/python-packaging-user-guide/blob/5b5c3b97f6a8871a5547c89c6962f105ea305403/source/namespace_packages.rst#pkg_resources-style-namespace-packages) – Eduardo Pignatelli Sep 04 '18 at 16:26
  • What is the example content of the module init module1/__init__.py? – openCivilisation May 25 '21 at 04:13
5

This section should be pretty self-explanatory.

In short, put the namespace code in __init__.py, update setup.py to declare a namespace, and you are free to go.

Asclepius
  • 57,944
  • 17
  • 167
  • 143
iElectric
  • 5,633
  • 1
  • 29
  • 31
2

This is an old question, but someone recently commented on my blog that my posting about namespace packages was still relevant, so thought I would link to it here as it provides a practical example of how to make it go:

https://web.archive.org/web/20150425043954/http://cdent.tumblr.com/post/216241761/python-namespace-packages-for-tiddlyweb

That links to this article for the main guts of what's going on:

http://www.siafoo.net/article/77#multiple-distributions-one-virtual-package

The __import__("pkg_resources").declare_namespace(__name__) trick is pretty much drives the management of plugins in TiddlyWeb and thus far seems to be working out.

Lars Francke
  • 716
  • 7
  • 18
cdent
  • 348
  • 1
  • 7
  • The siafoo link is broken, here's a link to an archived copy: https://web.archive.org/web/20200926015931/http://www.siafoo.net/article/77 – JCowfer Oct 28 '21 at 20:34
-11

You have your Python namespace concepts back to front, it is not possible in python to put packages into modules. Packages contain modules not the other way around.

A Python package is simply a folder containing a __init__.py file. A module is any other file in a package (or directly on the PYTHONPATH) that has a .py extension. So in your example you have two packages but no modules defined. If you consider that a package is a file system folder and a module is file then you see why packages contain modules and not the other way around.

So in your example assuming Package-1 and Package-2 are folders on the file system that you have put on the Python path you can have the following:

Package-1/
  namespace/
  __init__.py
  module1.py
Package-2/
  namespace/
  __init__.py
  module2.py

You now have one package namespace with two modules module1 and module2. and unless you have a good reason you should probably put the modules in the folder and have only that on the python path like below:

Package-1/
  namespace/
  __init__.py
  module1.py
  module2.py
Tendayi Mawushe
  • 25,562
  • 6
  • 51
  • 57
  • I'm talking about things like `zope.x` where a bunch of related packages are released as separate downloads. – joeforker Nov 04 '09 at 18:53
  • Ok, but what is the effect you are trying to achieve. If the folders containing related packages all on the PYTHONPATH the Python interpreter will find them for you with no extra effort on your part. – Tendayi Mawushe Nov 04 '09 at 19:00
  • 5
    If you add both Package-1 and Package-2 to PYTHONPATH, only Package-1/namespace/ will be seen by Python. – Søren Løvborg Jul 21 '11 at 21:10