Struggling with python's import mechanism

Question

I am an experienced java enterprise developer but very new to python enterprise development shop. I am currently, struggling to understand why some imports work while others don't.

Some background: Our dev team recently upgraded python from 3.6 to 3.10.5 and following is our package structure

src/
bunch of files (dockerfile, Pipfile, requrirements.txt, shell scripts, etc)
  package/
      __init__.py
      moduleA.py
      subpackage1/
          __init__.py
          moduleX.py
          moduleY.py
      subpackage2/
          __init__.py
          moduleZ.py
      tests/
          __init__.py
          test1.py
    

Now, inside the moduleA.py, I am trying to import subpackage2/moduleZ.py like so

from .subpackage2 import moduleZ

But, I get the error saying

ImportError: attempted relative import with no known parent package

The funny thing is that if I move moduleA.py out of package/ and into src/ then it is able to find everything. I am not sure why is this the case.

I run the moduleA.py by executiong python package/moduleA.py.

Now, I read that maybe there is a problem becasue you have you give a -m parameter if running a module as a script (or something on those lines). But, if I do that, I get the following error:

ModuleNotFoundError: No module names 'package/moduleA.py'

I even try running package1/moduleA and remove the .py, but that does not work either. I can understand why as I technically never installed it ?

All of this happened because apparently, the tests broke and to make it work they added relative imports. They changed the import from "from subpackage2 import moduleZ" to "from .subpackage2 import moduleZ" and the tests started working, but the app started failing.

Any understanding I can get would be much appreciated.

Answer 1

The -m parameter is used with the import name, not the path. So you'd use python3 -m package.moduleA (with . instead of /, and no .py), not python3 -m package/moduleA.py.

That said, it only works if package.moduleA is locatable from one of the roots in sys.path. Shy of installing the package, the simplest way to make it work is to ensure your working directory is src (so package exists in the working directory):

$ cd path/to/src
$ python3 -m package.moduleA

and, with your existing setup, if moduleA.py includes a from .subpackage2 import moduleZ, the import should work; Python knows package.moduleA is a module within package, so it can use a relative import to look for a sibling package to moduleA named subpackage2, and then inside it it can find moduleZ.

Obviously, this is brittle (it only works if you cd to the src root directory before running Python, or hack the path to src in PYTHONPATH, which is terrible hack if the code ever has to be run by anyone else); ideally you make this an installable package, install it (in global site-packages, user site-packages, or within a virtual environment created with the built-in venv module or the third-party virtualenv module), and then your working directory no longer matters (since the site-packages will be part of your sys.path automatically). For simple testing, as long as the working directory is correct (not sure what it was for you), and you use -m correctly (you were using it incorrectly), relative imports will work, but it's not the long term solution.

Answer 2

So first of all - the root importing directory is the directory from which you're running the main script.

This directory by default is the root for all imports from all scripts.

So if you're executing script from directory src you can do such imports:

from package.moduleA import *
from package.subpackage1.moduleX import *

But now in files moduleA and moduleX you need to make imports based on root folder. If you want to import something from module moduleY inside moduleX you need to do:

# this is inside moduleX
from package.subpackage1.moduleY import *

This is because python is looking for modules in specific locations. First location is your root directory - directory from which you execute your main script. Second location is directory with modules installed by PIP. You can check all directories using following:

import sys
for p in sys.path:
    print(p)

Now to solve your problem there are couple solutions. The fast one but IMHO not the best one is to add all paths with submodules to sys.path - list variable with all directories where python is looking for modules.

new_path = "/path/to/application/app/folder/src/package/subpackage1"
if new_path not in sys.path:
    sys.path.append(new_path)

Another solution is to use full path for imports in all package modules:

from package.subpackage1.moduleX import *

I think in your case it will be the correct solution.

You can also combine 2 solutions. First add folders with subpackages to sys.path and use subpackage folders as a root folders for imports. But it's good solution only if you have complex submodule structure. And it's not the best solution if in future you will need to deploy your package as a wheel or share between multiple projects.