Python can't find my module

Question

I have a python project (which I run within a virtualenv) and that has the following structure:

Project
├───.git
├───venv
└───src
    ├───__init__.py
    ├───mymodules
    │   ├───__init__.py
    │   ├───module1.py
    │   └───module2.py
    └───scripts
        ├───__init__.py
        └───script.py

script.py

import src.mymodules.module1
...

---

I run the project with venv activated and from the Project directory using the following command:

(venv)$ python src/scripts/script.py

The script runs but gives out the following error before exiting:

Traceback (most recent call last):
  File "src/scripts/script.py", line 1, in <module>
    import src.mymodules.module1
ImportError: No module named src.mymodules.module1

I have tried running the python shell and trying to import the module from there and it gave no errors. I have _ _init__.py in every directory within src. Is python considering the working directory to be src/scripts? Why is that happening and how can I make src the working directory if that's the case?

Answer 1

Essentially, when you execute script.py directly, it doesn't know that it's part of a submodule of src, nor does it know where a module named src might be. This is the case in either python 2 or 3.

As you know, Python finds modules based on the contents of sys.path. In order to import any module, it must either be located in a directory that's listed in sys.path, or, in the same directory as the script that you're running.

When you say python src/scripts/script.py, sys.path includes the Project/src/scripts/ (because that's where script.py is located), but not Project. Because Project isn't in the path, the modules in that directory (src) aren't able to be imported.

To fix this:

I'm assuming that your script.py is an entry point for your src module (for example, maybe it's the main program). If that's true, then you could fix it by moving script.py up to the same level as src:

Project
├───.git
├───venv
|───script.py       <--- script.py moves up here
└───src
    ├───__init__.py
    └───mymodules
        ├───__init__.py
        ├───module1.py
        └───module2.py

This way, script.py can freely import anything in src, but nothing in src can import script.py.

If that's not the case, and script.py really is a part of src, you can use python's -m argument to execute script.py as part of the src module like so:

$ python -m src.scripts.script

Because you've told python which module you're running (src), it will be in the path. So, script.py will be aware that it's a submodule of src, and then will be able to import from src.

Be careful in this situation though - there's potential to create a circular import if something in src imports src.scripts.script.

---

As an alternative to both of these approaches, you can modify the sys.path directly in script.py:

import sys
sys.path.insert(0, '/path/to/Project') # location of src 

While this works, it's not usually my preference. It requires script.py to know exactly how your code is laid out, and may cause import confusion if another python program ever tries to import script.py.

Answer 2

Project
├───.git
├───venv
└───src
    ├───__init__.py
    ├───mymodules
    │   ├───__init__.py
    │   ├───module1.py
    │   └───module2.py
    └───scripts
        ├───__init__.py
        └───script.py

Alternatively you can import like the following in your script.py

import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__),'../../'))
import src.mymodules.module1

Now you can run script.py file from any location.

e.g :
python script.py
python /path to folder/script.py

Answer 3

Another solution is creating file some_file.pth and write your project dir (mother dir of your src). Put this file in /virtual_env/lib/pythonXX/site-packages/. In this way, you don't need to import sys.path in your script.

Answer 4

If you face this problem when dealing with Pytest or coverage. Adding __init__.py file solve most of the cases.