2

I have a directory structure like this.

Chatbot/
   utils/
     abc.py
   projects/
      proj1/
        utils/
          __init__.py
          data_process.py
        components/
          class1.py

I have two utils folders in my structure, one at the top level and one inside my projects folder.

Now I want to import data_process.py file inside class1.py. So I tried like this

from utils.data_process import DataProcess

but it is referencing the top-level utils folder and even VSCode is not recognizing it. I tried creating __init__.py file inside the utils folder but still did not work.

I tried with empty __init__.py and then placing this content

from . import data_process
__all__ = ['data_proces']

then

from .data_process import DataPreprocess
__all__ = ['DataPreprocess']

then I tried

from ..utils.data_process import DataProcess

VSCode is recognizing this but it is not working and throws the error

ValueError: attempted relative import beyond top-level package

Even I tried changing the name utils to some other name but still the same issue

How can I solve this?

KetZoomer
  • 2,701
  • 3
  • 15
  • 43
Sunil Garg
  • 14,608
  • 25
  • 132
  • 189

2 Answers2

1

A python module is defined by a special file called __init__.py and this file should be present in all the subdirectories. So, your file structure would be:

Chatbot/
   __init__.py
   utils/
     __init__.py
     abc.py
   projects/
      __init__.py
      proj1/
        __init__.py
        utils/
          __init__.py
          data_process.py
        components/
          __init__.py
          class1.py
          class2.py

Now, you can do a relative import like:

  • Use . for importing something within the same directory. Example:
# file Chatbot/projects/proj1/components/class2.py
from .class1 import *
  • Similarly, use .. for two-level, and ... for three-level, and so on!

For instance:

# file Chatbot/projects/proj1/components/class2.py
from ..utils.data_process import * # import from data_process.py
# there will be four "." earlier I made a counting mistake
from ....utils.abc import * # import something from abc.py

Coding Convention

When writing a python module/package, you might want to follow PEP8 Convention.

Moreover, I believe you are trying to do different projects using your package Chatbot, is that correct? Then, it is a good practice that you set PYTHONPATH for Chatbot and do all the projects, and imports seperately.

EDIT: Uploading Dummy File

I'm able to work on this project seamlessly even with using two utils directories. Project Structure:

enter image description here

main file and its output:

# file main.py
from chatbot.utils.abc import hello as a1
from chatbot.projects.proj1.components.class1 import super_hello
from chatbot.projects.proj1.utils.data_process import hello as b1

print(a1(), b1())
print(super_hello())

enter image description here

Similarly, if you have chatbot under PYTHONPATH you can call the project from anywhere from your device.

Code details and files are added into my github account for details.

Mr. Hobo
  • 530
  • 1
  • 7
  • 22
  • chatbot outer project use the model created from inner project, thats why it contains projects folder.. projects folder contains the independent project but its output is used on top level chatbot project. – Sunil Garg Jul 28 '21 at 06:18
  • and do i need to mention PYTHONPTH in env variable? – Sunil Garg Jul 28 '21 at 06:19
  • and i need to add init.py file to every folder, that are not even packages? – Sunil Garg Jul 28 '21 at 06:21
  • I don't understand by outer/inner project that you have mentioned. You might want to look into these to clear concepts: Python Packaging - [[1](https://packaging.python.org/tutorials/packaging-projects/), [2](https://www.tutorialsteacher.com/python/python-package)]; Import UD Package - [[1](https://stackoverflow.com/questions/37516579), [2](https://www.youtube.com/watch?v=1RuMJ53CKds&ab_channel=Telusko)] – Mr. Hobo Jul 29 '21 at 05:45
  • outer project is chatbot and inner project chatbot/projects/proj1 – Sunil Garg Jul 29 '21 at 05:48
  • Technically, you should [modularize python code](https://www.python-course.eu/python3_modules_and_modular_programming.php) when you want to group similar functions. Does this outer and inner project is related? If yes, then you might want to combine them. – Mr. Hobo Jul 29 '21 at 05:51
  • 1
    If you are creating a `chatbot` and you want to use this **python package** in different codes, then this should not be the way you design the code. Instead you should use `PYTHONPATH` or `sys.path` mentioned by @SorousH Bakhtiary – Mr. Hobo Jul 29 '21 at 05:53
1

sys.path is where Python searches to find modules and packages and It does it in order.

So you can put ...../proj1/ at the beginning of the list, when python start searching it will find the utils folder in that path first !

import sys
sys.path.insert(0, r'...../proj1/')

But this cause another problem, python always find utils in that folder first, that's not what you want.

solution number 1

Absolute import:

from Chatbot.projects.proj1.utils.data_process import DataProcess

solution number 2

use relative imports which is mentioned by @Mr.Hobo.

S.B
  • 13,077
  • 10
  • 22
  • 49
  • Yes, `sys.path` method is good.. but it is tedious to mention each time... you can get over this using the `PYTHONPATH` [tutorial in windows](https://www.geeksforgeeks.org/pythonpath-environment-variable-in-python/) – Mr. Hobo Jul 29 '21 at 05:48
  • @Mr.Hobo emm In fact both of them(PYTHONPATH or sys.path) are causing problems. The main issue is we have two names called `utils` ! whichever path is accessible first in sys.path, that `utils` will be encountered first it doesn't matter where we are. So better solution is using absolute import as I mentioned. – S.B Jul 29 '21 at 08:44
  • 1
    please check the updated answer. I was able to do the project with ease w/o any problem. I have also added a link to my personal GitHub project account related to this project structure. – Mr. Hobo Jul 30 '21 at 06:10
  • 1
    @Mr.Hobo I've updated my answer and referenced to yours for relative imports. – S.B Jul 30 '21 at 17:08