Intra-package imports do not always work

Question

I have a Django project structured like so:

appname/
   models/
      __init__.py
      a.py
      base.py
      c.py

... where appname/models/__init__.py contains only statements like so:

from appname.models.base import Base
from appname.models.a import A
from appname.models.c import C

... and where appname/models/base.py contains:

import django.db.models


class Base(django.db.models.Model):
   ...

and where appname/models/a.py contains:

import appname.models as models


class A(models.Base):
   ....

...and similarly for appname/models/c.py, etc..

I am quite happy with this structure of my code, but of course it does not work, because of circular imports.

When appname/__init__.py is run, appname/models/a.py will get run, but that module imports "appname.models", which has not finished executing yet. Classic circular import.

So this supposedly indicates that my code is structured poorly and needs to be re-designed in order to avoid circular dependency.

What are the options to do that?

Some solutions I can think of and then why I don't want to use them:

1. Combine all my model code into a single file: Having 20+ classes in the same file is a far worse style than what I am trying to do (with separate files), in my opinion.
2. Move the "Base" model class into another package outside of "appname/models": This means that I would end up with package in my project that contains base/parent classes that should ideally be split into the packages in which their child/sub classes are located. Why should I have base/parent classes for models, forms, views, etc. in the same package and not in their own packages (where the child/sub classes would be located), other than to avoid circular imports?

So my question is not just how to avoid circular imports, but to do so in a way that is just as clean (if not cleaner) that what I tried to implement.

Does anyone have a better way?

Answer 1

Edit

I have researched this more thoroughly and come to the conclusion that this is a bug in either core Python or the Python documentation. More information is available at this question and answer.

Python's PEP 8 indicates a clear preference for absolute over relative imports. This problem has a workaround that involves relative imports, and there is a possible fix in the import machinery.

My original answer below gives examples and workarounds.

Original answer

The problem, as you have correctly deduced, is circular dependencies. In some cases, Python can handle these just fine, but if you get too many nested imports, it has issues.

For example, if you only have one package level, it is actually fairly hard to get it to break (without mutual imports), but as soon as you nest packages, it works more like mutual imports, and it starts to become difficult to make it work. Here is an example that provokes the error:

level1/__init__.py

    from level1.level2 import Base

level1/level2/__init__.py

    from level1.level2.base import Base
    from level1.level2.a import A

level1/level2/a.py

    import level1.level2.base
    class A(level1.level2.base.Base): pass

level1/level2/base

    class Base: pass

The error can be "fixed" (for this small case) in several different ways, but many potential fixes are fragile. For example, if you don't need the import of A in the level2 __init__ file, removing that import will fix the problem (and your program can later execute import level1.level2.a.A), but if your package gets more complex, you will see the errors creeping in again.

Python sometimes does a good job of making these complex imports work, and the rules for when they will and won't work are not at all intuitive. One general rule is that from xxx.yyy import zzz can be more forgiving than import xxx.yyy followed by xxx.yyy.zzz. In the latter case, the interpreter has to have finished binding yyy into the xxx namespace when it is time to retrieve xxx.yyy.zzz, but in the former case, the interpreter can traverse the modules in the package before the top-level package namespace is completely set up.

So for this example, the real problem is the bare import in a.py This could easily be fixed:

    from level1.level2.base import Base
    class A(Base): pass

Consistently using relative imports is a good way to enforce this use of from ... import for the simple reason that relative imports do not work without the from'. To use relative imports with the example above,level1/level2/a.py` should contain:

from .base import Base
class A(Base): pass

This breaks the problematic import cycle and everything else works fine. If the imported name (such as Base) is too confusingly generic when not prefixed with the source module name, you can easily rename it on import:

from .base import Base as BaseModel
class A(BaseModel): pass

Although that fixes the current problem, if the package structure gets more complex, you might want to consider using relative imports more generally. For example, level1/level2/__init__.py could be:

from .base import Base
from .a import A