cannot import submodule from a module

Question

I am having following structure for my project

Object_Detection/
                 setup.py
                 setup.cfg
                 requirement.txt
                 object_detection/
                                  models
                                  __init__.py #contains from . import models
                 tests/ 
                       # inside tests dir
                       test_utils_image.py
                       __init__.py #empty
                 utils/
                      # inside utils dir
                      __init__.py #inside 
                      utils_image_preprocess.py
                      utils_image.py
                      utils_tfrecord.py

Now init.py inside utils directory contains the following code.

# inside __init__.py
from . import utils_image_preprocess
from . import utils_image
from . import utils_tfrecord

Running above init.py files gives me an error:

ImportError: attempted relative import with no known parent package

test_utils.py inside tests dir contains the following code

# inside test_utils.py
from object_detection.utils import utils_image

While running test_utils.py I got the following error

ImportError: cannot import name 'utils_image' from 'object_detection.utils'

I have gone through this and this and tried to follow every aspect mentioned there but details about what to put inside init.py is not clear.

This problem seems to be associated with the structuring of init.py in different dir.

I have gone through various and got to know that if we keep even an empty init.py file then things will work out but I am not sure about my understanding.

Please let me know

1. what I am missing here and whether I am following the correct structure for packaging my code or not?
2. How to resolve these two errors?
3. Is this issue related to setting up source in IDE as I am using Vscode and I have also seen this has been mentioned at many places. See here? (But also tried the same code with PyCharm and encountered same error )

Answer 1

If you want to be able to say ...

from object_detection.utils import utils_image

... then clearly the utils directory must be a subdirectory of the object_detection directory and not a sibling directory, i.e. at the same level.

Now for your other error:

ImportError: attempted relative import with no known parent package

You did not really specify under what circumstances you get this error other than saying "Running above init.py files gives me an error:". But how are you "running" these py files or what does that even mean?

If you are executing a script when this occurs (how else would you be getting this error?), the script must be invoked as a module (because scripts cannot have relative imports -- see below) as follows (we will assume that the script you are trying to execute is test_utils_image.py):

First, the parent directory of object_detection, which is Object_Detection, must be in the system path of directories to be searched for finding modules and packages referenced in import statements. In general, this can be accomplished several ways, for instance

1. The script you are executing is in Object_Detection (the directory of the script is automatically added to the sys.path list of directories to be searched by the interpreter).
2. Dynamically appending Object_Detection to the sys.path list of directories at runtime by your script.
3. Appending Object_Detection to the PYTHONPATH environment variable.

Item 1 above would not be applicable for this specific case since the module we are executing by definition is not in the Object_Detection directory.

Note that if your classes will eventually be installed with pip, then site-packages will be the parent directory of object_detection, which is already in sys.path.

Then you can execute your script as:

python -m tests.test_utils_image

If you want to execute this .py file as a script, for example by right-mouse clicking on it is VS Code, then see Relative imports for the billionth time, in particular the section Scripts can't import relative, which says it all -- it cannot work!

To invoke this as a script, just convert the relative imports to absolute imports. In fact, the PEP 8 Style Guide says:

Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path):

Answer 2

Have you tried to do the following?

inside your utils __init__.py import your modules as follows:

from .utils_image_preprocess import <func1>... <rest of functions/classes you want to import>
from .utils_image import <func1>... <rest of functions/classes you want to import>
from .utils_tfrecord import <func1>... <rest of functions/classes you want to import>

And then in your test file do:

from object_detection.utils.utils_image import *

OR

from object_detection.utils.utils_image import <func1>,...

Also, make sure you don't have any circular dependencies in your modules. for example importing of function from your tests to your util module and vise versa

Answer 3

Python3 has two types of packages

1. Regular Packages
2. Namespace Packages

Regular packages contains init.py and namespace packages don't need to have init.py

Regarding your folder structure, it is correct, no change needed. You just need to import like this

from utils import utils_image

without mentioning the objects_detection as objects_detection is just a namespace package.

So it would be usefull when you would refer to the utils module from outside the objects_detection folder.

Here what python docs say about the namespace packages:

A namespace package is a composite of various portions, where each portion contributes a subpackage to the parent package.