nightmare with relative imports, how does pep 366 work?

Question

I have a "canonical file structure" like that (I'm giving sensible names to ease the reading):

mainpack/

  __main__.py
  __init__.py 

  - helpers/
     __init__.py
     path.py

  - network/
     __init__.py
     clientlib.py
     server.py

  - gui/
     __init__.py
     mainwindow.py
     controllers.py

In this structure, for example modules contained in each package may want to access the helpers utilities through relative imports in something like:

# network/clientlib.py
from ..helpers.path import create_dir

The program is runned "as a script" using the __main__.py file in this way:

python mainpack/

Trying to follow the PEP 366 I've put in __main__.py these lines:

___package___ = "mainpack"
from .network.clientlib import helloclient 

But when running:

$ python mainpack 
Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "path/mainpack/__main__.py", line 2, in <module>
    from .network.clientlib import helloclient
SystemError: Parent module 'mainpack' not loaded, cannot perform relative import

What's wrong? What is the correct way to handle and effectively use relative imports?

I've tried also to add the current directory to the PYTHONPATH, nothing changes.

Answer 1

The "boilerplate" given in PEP 366 seems incomplete. Although it sets the __package__ variable, it doesn't actually import the package, which is also needed to allow relative imports to work. extraneon's solution is on the right track.

Note that it is not enough to simply have the directory containing the module in sys.path, the corresponding package needs to be explicitly imported. The following seems like a better boilerplate than what was given in PEP 366 for ensuring that a python module can be executed regardless of how it is invoked (through a regular import, or with python -m, or with python, from any location):

# boilerplate to allow running as script directly
if __name__ == "__main__" and __package__ is None:
    import sys, os
    # The following assumes the script is in the top level of the package
    # directory.  We use dirname() to help get the parent directory to add to
    # sys.path, so that we can import the current package.  This is necessary 
    # since when invoked directly, the 'current' package is not automatically
    # imported.
    parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    sys.path.insert(1, parent_dir)
    import mypackage
    __package__ = str("mypackage")
    del sys, os

# now you can use relative imports here that will work regardless of how this
# python file was accessed (either through 'import', through 'python -m', or 
# directly.

If the script is not at the top level of the package directory and you need to import a module below the top level, then the os.path.dirname has to be repeated until the parent_dir is the directory containing the top level.

Answer 2

Inspired by extraneon's and taherh's answers here is some code that runs up the file tree until it runs out of __init__.py files to build the full package name. This is definitely hacky, but does seem to work regardless of the depth of the file in your directory tree. It seems absolute imports are heavily encouraged.

import os, sys
if __name__ == "__main__" and __package__ is None:
    d,f = os.path.split(os.path.abspath(__file__))
    f = os.path.splitext(f)[0]
    __package__ = [f] #__package__ will be a reversed list of package name parts
    while os.path.exists(os.path.join(d,'__init__.py')): #go up until we run out of __init__.py files
        d,name = os.path.split(d) #pull of a lowest level directory name 
        __package__.append(name)  #add it to the package parts list
    __package__ = ".".join(reversed(__package__)) #create the full package name
    mod = __import__(__package__) #this assumes the top level package is in your $PYTHONPATH
    sys.modules[__package__] = mod  #add to modules 

Answer 3

The loading code seems to be something like this:

    try:
        return sys.modules[pkgname]
    except KeyError:
        if level < 1:
            warn("Parent module '%s' not found while handling "
                 "absolute import" % pkgname, RuntimeWarning, 1)
            return None
        else:
            raise SystemError, ("Parent module '%s' not loaded, cannot "
                                "perform relative import" % pkgname)

which makes me think that maybe your module is not on sys.path. If you start Python (normally) and just type "import mainpack" on the prompt, what does it do? It should be able to find it.

I have tried it myself and got the same error. After reading a bit I found the following solution:

# foo/__main__.py
import sys
mod = __import__('foo')
sys.modules["foo"]=mod

__package__='foo'
from .bar import hello

hello()

It seems a bit hackish to me but it does work. The trick seems to be making sure package foo is loaded so the import can be relative.

Answer 4

This is a minimal setup based on most of the other answers, tested on python 2.7 with a package layout like so. It also has the advantage that you can call the runme.py script from anywhere and it seems like it's doing the right thing - I haven't yet tested it in a more complex setup, so caveat emptor... etc.

This is basically Brad's answer above with the insert into sys.path others have described.

packagetest/
  __init__.py       # Empty
  mylib/
    __init__.py     # Empty
    utils.py        # def times2(x): return x*2
  scripts/
    __init__.py     # Empty
    runme.py        # See below (executable)

runme.py looks like this:

#!/usr/bin/env python
if __name__ == '__main__' and __package__ is None:
    from os import sys, path
    d = path.dirname(path.abspath(__file__))
    __package__ = []
    while path.exists(path.join(d, '__init__.py')):
        d, name = path.split(d)
        __package__.append(name)
    __package__ = ".".join(reversed(__package__))
    sys.path.insert(1, d)
    mod = __import__(__package__)
    sys.modules[__package__] = mod

from ..mylib.utils import times2

print times2(4)