394

I am using Python 3.5.1. I read the document and the package section here: https://docs.python.org/3/tutorial/modules.html#packages

Now, I have the following structure:

/home/wujek/Playground/a/b/module.py

module.py:

class Foo:
    def __init__(self):
        print('initializing Foo')

Now, while in /home/wujek/Playground:

~/Playground $ python3
>>> import a.b.module
>>> a.b.module.Foo()
initializing Foo
<a.b.module.Foo object at 0x100a8f0b8>

Similarly, now in home, superfolder of Playground:

~ $ PYTHONPATH=Playground python3
>>> import a.b.module
>>> a.b.module.Foo()
initializing Foo
<a.b.module.Foo object at 0x10a5fee10>

Actually, I can do all kinds of stuff:

~ $ PYTHONPATH=Playground python3
>>> import a
>>> import a.b
>>> import Playground.a.b

Why does this work? I though there needed to be __init__.py files (empty ones would work) in both a and b for module.py to be importable when the Python path points to the Playground folder?

This seems to have changed from Python 2.7:

~ $ PYTHONPATH=Playground python
>>> import a
ImportError: No module named a
>>> import a.b
ImportError: No module named a.b
>>> import a.b.module
ImportError: No module named a.b.module

With __init__.py in both ~/Playground/a and ~/Playground/a/b it works fine.

yacc
  • 2,915
  • 4
  • 19
  • 33
wujek
  • 10,112
  • 12
  • 52
  • 88

5 Answers5

483

Overview

@Mike's answer is correct but too imprecise. It is true that Python 3.3+ supports Implicit Namespace Packages that allows it to create a package without an __init__.py file. This is called a namespace package in contrast to a regular package which does have an __init__.py file (empty or not empty).

However, creating a namespace package should ONLY be done if there is a need for it. For most use cases and developers out there, this doesn't apply so you should stick with EMPTY __init__.py files regardless.

Namespace package use case

To demonstrate the difference between the two types of python packages, lets look at the following example:

google_pubsub/              <- Package 1
    google/                 <- Namespace package (there is no __init__.py)
        cloud/              <- Namespace package (there is no __init__.py)
            pubsub/         <- Regular package (with __init__.py)
                __init__.py <- Required to make the package a regular package
                foo.py

google_storage/             <- Package 2
    google/                 <- Namespace package (there is no __init__.py)
        cloud/              <- Namespace package (there is no __init__.py)
            storage/        <- Regular package (with __init__.py)
                __init__.py <- Required to make the package a regular package
                bar.py

google_pubsub and google_storage are separate packages but they share the same namespace google/cloud. In order to share the same namespace, it is required to make each directory of the common path a namespace package, i.e. google/ and cloud/. This should be the only use case for creating namespace packages, otherwise, there is no need for it.

It's crucial that there are no __init__py files in the google and google/cloud directories so that both directories can be interpreted as namespace packages. In Python 3.3+ any directory on the sys.path with a name that matches the package name being looked for will be recognized as contributing modules and subpackages to that package. As a result, when you import both from google_pubsub and google_storage, the Python interpreter will be able to find them.

This is different from regular packages which are self-contained meaning all parts live in the same directory hierarchy. When importing a package and the Python interpreter encounters a subdirectory on the sys.path with an __init__.py file, then it will create a single directory package containing only modules from that directory, rather than finding all appropriately named subdirectories outside that directory. This is perfectly fine for packages that don't want to share a namespace. I highly recommend taking a look at Traps for the Unwary in Python’s Import System to get a better understanding of how Python importing behaves with regular and namespace package and what __init__.py traps to watch out for.

Summary

  • Only skip __init__.py files if you want to create namespace packages. Only create namespace packages if you have different libraries that reside in different locations and you want them each to contribute a subpackage to the parent package, i.e. the namespace package.
  • Keep on adding empty __init__.py to your directories because 99% of the time you just want to create regular packages. Also, Python tools out there such as mypy and pytest require empty __init__.py files to interpret the code structure accordingly. This can lead to weird errors if not done with care.

Resources

My answer only touches the surface of how regular packages and namespace packages work, so take a look at the following resources for further information:

hakre
  • 193,403
  • 52
  • 435
  • 836
Andi
  • 8,154
  • 3
  • 30
  • 34
  • 2
    Suppose I have `run_script.py` in same dir as `parent_package` so can I just import like `from parent_package.child_package import child1` without `__init__.py` ? – mrgloom May 16 '18 at 15:10
  • Is the purpose of this so you can write child_package.some_function even if some_function is defined in childX.py? In another words it avoids requiring the user to know about the different files in child_package? ? – johnbakers Jun 25 '18 at 13:29
  • Yeah, I don’t get why you would make `child1.py`, `child2.py` instead of just putting their code together into `__init__`.py directly. – binki Nov 26 '18 at 00:10
  • Shouldn't the import statements in `__init__` be relative imports i.e. `from . import child1`? The absolute import gives me `ModuleNotFoundError` (in Python 3.6) – Halbeard Feb 03 '19 at 13:34
  • 18
    In my experience, even with python 3.3+, an empty `__init__.py` is still needed sometimes, like when you want to refer a subfolder as a package. For example, if I run `python -m test.foo` it didn't work until I created an empty `__init__.py` under the test folder. And I'm talking about 3.6.6 version here! – Prahlad Yeri Jun 18 '19 at 16:20
  • 3
    @PrahladYeri This deserves to be an answer. – Flimm Nov 27 '20 at 17:01
  • Python looks like a silly language when **not** creating a file still works exactly the same but is supposed to be a conscious, informed choice. – Marcos Pereira Feb 01 '23 at 16:58
  • 2
    It makes more sense to only add a `__init__.py` if you need it. Python looks like a silly language when not creating a file still works exactly the same but is supposed to be a conscious, informed choice. – Marcos Pereira Feb 01 '23 at 17:45
  • 2
    This argument strikes me as bizarre. "You should only do nothing if you have a good reason why you need to do nothing, otherwise you can get away with doing something"... ? "I highly recommend taking a look at Traps for the Unwary in Python’s Import System" - this link advises (correctly) that a missing `__init__.py` only caused an error in 3.2 and below, and that adding `__init__.py` reduces flexibility (can be a plus or a minus, but I can hardly see a reason to do it preemptively). – Karl Knechtel Feb 06 '23 at 07:28
278

Python 3.3+ has Implicit Namespace Packages that allow it to create a packages without an __init__.py file.

Allowing implicit namespace packages means that the requirement to provide an __init__.py file can be dropped completely, and affected ... .

The old way with __init__.py files still works as in Python 2.

martineau
  • 119,623
  • 25
  • 170
  • 301
Mike Müller
  • 82,630
  • 20
  • 166
  • 161
  • 16
    I will read the document, but it's a bit long. Is it possible to quickly summarize? Could you just tell me: does it still support __init__.py, or completely ignores them? If it does support them, what is the difference in functionality and why this duality? – wujek May 10 '16 at 13:38
  • 1
    Yes, it still works. So all your Python 2 packages with `__init__.py` files will work in terms of imports (other difference between Python 2 and 3 not considered here). – Mike Müller May 10 '16 at 13:41
  • 3
    So the tutorial should be probably updated. Is a documentation bug opened for it? – Michel Samia Dec 27 '17 at 15:22
  • 13
    I'm still upset that this defies the `Zen Of Python` line 2: `Explicit is better than implicit.` .... – JayRizzo Jun 11 '18 at 06:23
  • 14
    @JayRizzo But: "Although practicality beats purity." – Mike Müller Jun 11 '18 at 06:58
  • 48
    @JayRizzo IMO it is even more explicit. Sometimes it happens to do init stuff in `__init__.py`, sometimes not. In Python 3 when I need these stuff I create a new `__init__.py` with specific code, otherwise I don't. This comes handy to know, visually, which packages have custom init. Instead in python 2 I always have to place an `__init__.py` (often empty), making a great number of them and finally harder to remember where you placed your init code. This should also fit "There should be one-- and preferably only one --obvious way to do it.". – Paolo Jul 14 '18 at 18:22
  • 1
    Is `__init__.py` is used in any case in Python 3? – mrgloom Feb 01 '19 at 09:39
  • 5
    Yes, there are several uses. One of them would "flattening the name space", i.e. importing modules from sub-packages to make them available directly under package. For a large example look a NumPy https://github.com/numpy/numpy/blob/master/numpy/__init__.py Here `from .core import *`imports all names from `core` and they can be accessed via `numpy.name`, where 'name' actually lives in 'numpy.core`. – Mike Müller Feb 01 '19 at 10:07
  • This is the actual documentation of how importing works and what built-in variables are available: https://docs.python.org/3/reference/import.html – jciloa Sep 04 '19 at 15:48
  • 5
    According to https://dev.to/methane/don-t-omit-init-py-3hga, you should not omit `__init__.py` in your project unless you are 100% sure about what "namespace package" is. – mzjn Jun 30 '20 at 09:45
  • 5
    See also https://packaging.python.org/guides/packaging-namespace-packages/ "Namespace packages allow you to split the sub-packages and modules within a single package across multiple, separate distribution packages (referred to as distributions in this document to avoid ambiguity)." Namespace package is not a regular package. `__init__.py` is not an old way. It is the only right way to create a regular package. – methane Jul 19 '20 at 04:45
  • It seems that when I import a package without the __init__.py everything is imported. Even packages imported by the module I am importing seem to be imported to the end module. – ed22 Jul 22 '20 at 23:18
32

If you have setup.py in your project and you use find_packages() within it, it is necessary to have an __init__.py file in every directory for packages to be automatically found.

Packages are only recognized if they include an __init__.py file

UPD: If you want to use implicit namespace packages without __init__.py you just have to use find_namespace_packages() instead

Docs

techkuz
  • 3,608
  • 5
  • 34
  • 62
17

I would say that one should omit the __init__.py only if one wants to have the implicit namespace package. If you don't know what it means, you probably don't want it and therefore you should continue to use the __init__.py even in Python 3.

Mi-La
  • 685
  • 10
  • 18
  • 4
    I think it's the other way around because there is a trap related to init with Python 3.3+. Wouldn't it be cleaner to just not have to init if it is empty? Especially if you have people coming from other languages. A question regarding why an empty init exists gets brought up often. If you have a specific need for init then one should use it imo. – NomNomNom Sep 29 '20 at 23:47
  • 1
    I guess that the rules come from older versions of Python and changing it dramatically would break all the backward compatibility. I had also problems to understand how the `__init__.py` works and I don't really like it, but we must live with what we get :-). Also note that there are also still some tools that expect `__init__.py` to be present in each package to work correctly. I remember that even pylint had some problems to implement implicit namespace packages logic correctly. – Mi-La Sep 30 '20 at 07:21
14

Based on my experience, even with python 3.3+, an empty __init__.py is still needed sometimes. One situation is when you want to refer a subfolder as a package. For example, when I ran python -m test.foo, it didn't work until I created an empty __init__.py under the test folder. And I'm talking about 3.6.6 version here which is pretty recent.

Apart from that, even for reasons of compatibility with existing source code or project guidelines, its nice to have an empty __init__.py in your package folder.

Prahlad Yeri
  • 3,567
  • 4
  • 25
  • 55
  • 2
    Can confirm. Using pytest 6.2.2 and python 3.8, I get a `AttributeError: 'NoneType' object has no attribute 'endswith'` when running `pytest . --doctest-modules`. The error disappeared after I added an empty `__init__.py` to one of my directories containing a python file containing doctests. It's NOT necessary to add that file to a different directory also containing a python file containing doctests. I don't understand it at all. – iron9 Jan 27 '21 at 16:25