3

This is my current directory structure, I am trying to import a function from src/helpers/log.py to src/data/download_dataset.py. I have followed this answer but it still does not work.

|-- AUTHORS.rst
|-- CONTRIBUTING.rst
|-- HISTORY.rst
|-- LICENSE
|-- MANIFEST.in
|-- Makefile
|-- README.rst
|-- data
|   |-- external
|   |-- interim
|   |-- processed
|   `-- raw
|       `-- wine-quality.csv
|-- docs
|   |-- Makefile
|   |-- authors.rst
|   |-- conf.py
|   |-- contributing.rst
|   |-- history.rst
|   |-- index.rst
|   |-- installation.rst
|   |-- make.bat
|   |-- readme.rst
|   `-- usage.rst
|-- dvc_mlflow
|   |-- __init__.py
|   `-- dvc_mlflow.py
|-- logs
|-- models
|-- requirements_dev.txt
|-- setup.cfg
|-- setup.py
|-- src 
|   |-- data
|   |   |-- __init__.py
|   |   `-- download_dataset.py
|   |-- features
|   |   `-- __init__.py
|   |-- helpers
|   |   |-- __init__.py
|   |   `-- log.py
|   `-- models
|       `-- __init__.py
|-- tests
|   |-- __init__.py
|   `-- test_dvc_mlflow.py
`-- tox.ini

I am importing the file log_error in src/data/download_dataset.py like so:

from helpers.log import log_error

But when I try to run the file using python3 src/data/download_dataset.py I get the error ModuleNotFoundError: No module named 'helpers'. I am a bit confused because I already added in the __init__.py files in each of the directories to make them modules but the issue still persists.

yudhiesh
  • 6,383
  • 3
  • 16
  • 49
  • You are running it from the root directory of the project which only has `src` in it. So it either needs to be `from src.helpers.log import log_error` or you need to run it from `src` directory. – Yevhen Kuzmovych Jul 20 '21 at 14:50

3 Answers3

2

You can try the sys.path.append method. Whatever modules you want to import, find the path to those modules and pass it to the function.

Example:

If my current working directory is /home/user_name/Desktop/Scripts/Main.py and I want to import some file Factorial.py which is at /home/user_name/Documents/OtherScripts/, I can do the following

# Inside your Main.py file
import sys
sys.path.append("/home/user_name/Documents/OtherScripts/")
from Factorial import *
Sai Suman Chitturi
  • 402
  • 2
  • 15
  • This works I used `sys.path.append("/Users/yravindranath/dvc_mlflow/")` but it seems rather hacky, isn't there another way without having to append the path? – yudhiesh Jul 20 '21 at 15:02
  • I am not sure about the correct way of doing it. Yeah, it's hacky. But it works :) – Sai Suman Chitturi Jul 20 '21 at 15:08
  • @yudhiesh Typically I do `sys.path.append`s with relative paths determined automatically. Your main file is in this directory `d = os.path.dirname(__file__)` so from there you can add `f"{d}/folder1/folder2"` or even go backwards `f"{d}/../parent folder"`. That way your code can be moved around and always work without having to do manuals adjustments to the paths as long as the general structure of the project is intact. – Guimoute Jul 24 '21 at 06:56
  • @Guimoute thanks for pointing that out, in that case then this answer is more suitable than mine. – yudhiesh Jul 24 '21 at 13:58
0

You are running it from the root directory of the project which only has src in it. So it either needs to be from src.helpers.log import log_error or you need to run it from src directory.

In general, you'd want to name your src directory with the same name as your project, so imports look logical.

Yevhen Kuzmovych
  • 10,940
  • 7
  • 28
  • 48
  • When I try `python3 download_dataset.py` and `from helpers.log import log_error` I get the same error. When I run `from src.helpers.log import log_error ` and running `python3 src/data/download_dataset.py` I get `No module named 'src'`. I updated the question with the full dir structure. – yudhiesh Jul 20 '21 at 14:53
  • Now you are running it from `data` directory. Try running it from the root directory – Yevhen Kuzmovych Jul 20 '21 at 14:54
  • What python version do you use? You might also need to add `__init__.py` to the `src` directory. – Yevhen Kuzmovych Jul 20 '21 at 14:58
  • I am using Python 3.8.8 but that does not solve it as well. – yudhiesh Jul 20 '21 at 15:00
0

I found an even simpler way that did not resort to me having to append the path using sys.path.append(), just needed to set the python path in the command line using export PYTHONPATH="${PYTHONPATH}:/path/to/your/project/".

yudhiesh
  • 6,383
  • 3
  • 16
  • 49