Rules for when recursive package imports fail

Question

This came up in the context of answering another question today.

Assume the following files, where the comments denote the filenames:

# level1/__init__.py
    import level1.level2
    answer = level1.level2.answer

# level1/level2/__init__.py
    from .b import answer

# level1/level2/b.py
    from .a import answer
    from ..level2.a import answer
    from level1.level2.a import answer
    import level1.level2.a

    if answer != 42:
        answer = level1.level2.a.answer  # <-- Fails here

#level1/level2/a.py
    answer = 42

With this code, python -c "import level1" works fine (in both 2.7 and 3.4) as-is. But change the answer to anything besides 42, and it fails with AttributeError: 'module' object has no attribute 'level2' at the designated location.

It appears that from/import can import a sub-module of a package and pull variables out of its namespace before all the parent namespaces are set up, but (obviously) the parent namespaces must be set up before the attributes of sub-namespaces can be traversed via normal attribute accesses.

Sometimes, though, the namespace is set up "well enough" for the import to work. For example, the code below, with the top level stripped off, always works with python -c "import level2", even though we still haven't finished initializing the level2 namespace when we import level2.a from level2.b.

# level2/__init__.py
    from .b import answer

# level2/b.py
    from .a import answer
    import level2.a

    if answer != 42:
        answer = level2.a.answer  # <-- Works here

#level1/level2/a.py
    answer = 41

EDIT

It appears that import x.y.z will insert a reference to z into y, but will not insert a reference to y into x. For example, when I change level1/level2/b.py from my first example:

# level1/level2/b.py
    from sys import modules

    def dump():
        print '\n Dumping...'
        for key, value in sorted(modules.items()):
            if key.startswith('level'):
                print key, [ x for x in dir(value) if not x.startswith('_')]

    dump()
    import level1.level2.a
    dump()

    from .a import answer
    from ..level2.a import answer
    from level1.level2.a import answer

    if answer != 42:
        answer = level1.level2.a.answer

I get the following results before the traceback:

 Dumping...
level1 []
level1.level2 []
level1.level2.b ['dump', 'modules']
level1.level2.sys []

 Dumping...
level1 []
level1.level2 ['a']
level1.level2.a ['answer']
level1.level2.b ['dump', 'level1', 'modules']
level1.level2.level1 []
level1.level2.sys []

However, if, right after the second call to dump(), I add the line:

setattr(modules['level1'], 'level2', modules['level1.level2'])

then it will not fail, because I have bound level2 into level1 before accessing it via attribute lookup. However, it seems that if the interpreter can bind a into level1 on that import, it might also be able to do the binding of level2 into level in exactly the same fashion.

Is there a good reason for why only the lowest-level package is updated during the import statement, or is it a bug (or perhaps, rather, a feature that was added for a single package that should be extended to nested packages)?

NOTE According to the Python documentation, When a submodule is loaded using any mechanism ... a binding is placed in the parent module’s namespace to the submodule object..

I believe this happens. But not soon enough in all cases. Also, the documentation for it may be relatively new.

Answer 1

I have filed an issue against Python. The text of this issue is reproduced below:

PEP 8 recommends absolute imports over relative imports, and section 5.4.2 of the import documentation says that an import will cause a binding to be placed in the imported module's parent's namespace.

However, since (with all current Python versions) this binding is not made until after the module body has been executed, there are cases where relative imports will work fine but absolute imports will fail. Consider the simple case of these five files:

xyz.py: import x
x/__init__.py:  import x.y
x/y/__init__.py:  import x.y.a
x/y/a/__init__.py:  import x.y.b; foo = x.y.b.foo
x/y/b/__init__.py:  foo = 1

This will fail in a fashion that may be very surprising to the uninitiated. It will not fail on any of the import statements; rather it will fail with an AttributeError on the assignment statement in x.y.a, because the import of y has not yet finished, so y has not yet been bound into x.

This could conceivably be fixed in the import machinery by performing the binding before performing the exec. Whether it can be done cleanly, so as not to cause compatibility issues with existing loaders, is a question for core maintainers.

But if it is decided that the current behavior is acceptable, then at a minimum both the PEP 8 and the import documentation should have an explanation of this corner case and how it can be solved with relative imports.