3

Imagine the following scenario. You're writing some module consisting of multiple files. As you're writing the code, eventually, you arrive at a situation where multiple files (let's say main.py and side.py) import each other, leading to a recursive import, a no-go.

module/
  main.py
  side.py

You decide to split main into multiple files: base.py and advanced.py. The former contains merely the basic definitions from main.py and does not make use of side nor any other submodules; other submodules which wish to make use of main should be satisfied with importing this. The latter (advanced.py) can freely import anything from side, but since side does not make use of advanced, there is no recursion. This resolves the recursive import.

Now, you're left with the following package structure:

module/
  side.py
  base.py
  advanced.py

It makes sense to put base and advanced into a subfolder main (thus creating a submodule), because this at least partially preserves the original package structure. Thus we obtain

module/
  side.py
  main/
    base.py
    advanced.py

But now consider a third file third.py, which originally imported the entire main:

from .main import *

The interface to main was broken by the aforementioned "recursion-fixing" operator. So, how does one restore the original interface, that is, how does one make from .main import * import everything from both base and advanced?

Example

Original main.py:

from .side import *

class A:
  pass

class B(C):
  pass

Original side.py:

from .main import *

class C:
  pass

class D(A):
  pass

After restructuring, main is split into two files:

# base.py
class A:
  pass

# advanced.py
from module.side import *

class B(C):
  pass

And side should now be importing main.base instead of main:

# new side.py
from .main.base import *

class C:
  pass

class D(A):
  pass

Here are some not totally satisfactory solutions/ideas:

__init__.py shenaningans

Create __init__.py in the main submodule, and place the following into it (solution from Implicit import from submodules):

from .base import *
from .advanced import *

The problem with this is that again introduces a recursive import: recall that side makes use of main.base, thus somewhere there, there is a line of code

from .main.base import *

which would invoke the __init__.py, causing side to import from advanced as well. Which we wanted to avoid. More specifically, putting the following into the python interpreter

from module.side import *

outputs the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../module/side.py", line 2, in <module>
    from .main.base import *
  File ".../module/main/__init__.py", line 2, in <module>
    from .advanced import *
  File ".../module/main/advanced.py", line 4, in <module>
    class B(C):
NameError: name 'C' is not defined

The reason being that, when loading advanced, it attempts to load side. But since side is "already loaded" (or rather is currently being loaded), to avoid infinite recursion, python merely skips over it. But then class C, which is required by advanced, is not loaded.

Instead put the shenaningans in all.py

Instead of putting the aforementioned two lines of code inside a __init__.py, put it in another file all.py. Now, when one wants to import everything from main, one writes

from .main.all import *

This is still not the same as from .main import *, so whenever such "recursion-fixing" restructuring of the package takes place, one would have to look for all such imports and rewrite them (by appending .all).

buj
  • 165
  • 5
  • I would like to know after two and a half year if you have found a solution for yourself now? – buhtz Mar 04 '22 at 08:09
  • no, I think I just accepted (as a user of python) that some stuff just doesn't exist – buj Mar 05 '22 at 16:36

1 Answers1

1

In many cases, this method works. that you import the module's object inside of the block, for example, if side.py uses main.A and main.py uses side.C, You can define the simple function in each module and import A or C inside it, and the function return C or A class.

in main.py:

def get_C():
    from .side import C
    return C

class A:
  pass

class B(get_C()):
  pass

and in side.py:

def get_A():
    from .main import A
    return A

class C:
  pass

class D(get_A()):
  pass

In this situation, circular import error does not occur

kamyarmg
  • 760
  • 2
  • 6
  • 23