Python: Module Import Error for the billionth time

Question

I can't believe after 8 years and 10,000 hours of using Python that I still get stumped by module import errors!!!! My tree is as follows:

pcode
    /general
    /latin
        /latin
            v_reading.py
    /other
        add_path.py
        use_pydub.py

In the add_path.py module I add some modules to the sys.path like so:

import sys

vol='/users/me/'
gen_dir = f'{vol}Documents/pcode/'
sys.path.append(gen_dir)
gen_dir1 = f'{vol}Documents/pcode/byu_corpus/'
sys.path.append(gen_dir1)
gen_dir1 = f'{vol}Documents/pcode/latin/'
sys.path.append(gen_dir1)
gen_dir1 = f'{vol}Documents/pcode/latin/latin/'
sys.path.append(gen_dir1)
gen_dir1 = f'{vol}Documents/pcode/general/'
sys.path.append(gen_dir1)
gen_dir1 = f'{vol}Documents/pcode/other/'
sys.path.append(gen_dir1)

so when I run sys.path I get inter alia:

/Users/me/Documents/pcode/other
/users/me/documents/pcode/
/users/me/documents/pcode/latin/   ## I'm not sure if this is necessary but I did it anyway
/users/me/documents/pcode/latin/latin/
/users/me/documents/pcode/general/
/users/me/documents/pcode/other/

The first lines of the use_pydub.py module are as follows:

import sys
import add_path
from general import *
import latin
from latin.latin.v_reading import focus

I do not get an error when I run the module from pycharm but I do get the following error when I run it from command line:

ModuleNotFoundError: No module named 'latin.latin'

I don't understand why I can import the 'latin' folder but not the 'latin.latin' folder. I also tried renaming the second latin folder to latin2 but that did not help. I should also point out that there is no difference between the sys.path generated by pycharm and the sys.path generated by command line.

does your `latin/latin` folder contain an `__init__.py` file? — Valentino, Aug 28 '22 at 16:39
Kinda hard to see that folder structure when is not formatted — Mad Physicist, Aug 28 '22 at 16:41
Right after `import latin`, add `print(latin.__file__)`. Is it the path you want? You've added multiple directories that either have a subdirectory called "latin" or perhaps a module named "latin.py". Which one is `import latin` supposed to pick? It seems like you have a package, so only put that package's directory in sys.path, not all of its subpackage directories. — tdelaney, Aug 28 '22 at 16:43
Remove the code adding `Documents/pcode/latin/latin/` and you should be all set. — MattDMo, Aug 28 '22 at 16:54
Today it [is recommend](https://packaging.python.org/en/latest/tutorials/packaging-projects/) to use the `src` layout for your project folder. Please see here for an example: https://stackoverflow.com/a/73166679/4865723 — buhtz, Aug 28 '22 at 16:56
I might be wrong here, but it seems to me you only want `pcode` is your `sys.path`, and none of its subfolders. When you do an import like `latin.latin`, it looks for a directory called `latin` in a folder in your `sys.path`, and then a directory called `latin` or a file called `latin.py` inside that `latin` folder. So neither `pcode/latin` nor `pcode/latin/latin` belong in your `sys.path`. — joanis, Aug 28 '22 at 16:56
Python treats the top level executing script (always called `__main__`) differently than any modules or packages that are imported. So, how do you run `add_path.py`? Is it a top level script or is it imported? Scripts are not a part of packages. It looks like `pcode` is a package. Generally, you'd put pcode's parent in sys.path (or better yet, write a setup.py and install the thing). In your case, thats /Users/me/Documents, which is not a good candidate for a package directory. You likely need some restructuring. — tdelaney, Aug 28 '22 at 16:58
Instead of trying to hack `sys.path` (and requiring everyone to remember to import the path-adding module before anything else important... and what if you can't figure out how to import `add_path` from the current module?), please just *learn to use relative imports properly*. This is *especially* important if you are trying to package the program and make it installable from PyPI, because those paths **will not be valid on someone else's computer!** Imports in Python are *really not that hard*. — Karl Knechtel, Aug 29 '22 at 04:05

score 1 · Answer 1 · answered Aug 29 '22 at 04:14

Solve the problem by correctly using relative imports:

Delete add_path.py, as it will no longer be useful or relevant.

In use_pydub.py, the appropriate relative imports look like:

import sys
from ..general import *
from .. import latin
from ..latin.latin.v_reading import focus

Ensure that pcode is a top-level package when the code runs. If, for example, use_pydub.py is intended to be a top-level script, run it as a module instead, with the -m flag to Python. Alternately, have a top-level driver script that uses the pcode package appropriately.
Ensure that the package is on the module import path. This doesn't require hacking sys.path, in general. It requires starting the program from just outside pcode, in development; or you can simply rely on the fact that installing the package, per the usual process will put it into one of the default paths where Python looks for third-party libraries. (That's the point of all that packaging stuff, using PyPI and pip etc.)

These things are especially important if you want to package the code and upload it to PyPI for others to use. Absolute paths hacked into sys.path are not going to work on someone else's computer - with a different username, the code installed into who knows which Python installation located who knows where on the disk, etc. Hacking in relative paths is just clumsily reinventing the wheel.

score 0 · Answer 2 · answered Aug 29 '22 at 03:51

Problem solved. By using the advice of tdelaney I ran print(latin.__file__) I discovered that I was not importing the correct package and that I had earlier downloaded my own package from pypi and placed that package in the site_packages and one of those packages was named latin. That packaged was interfering with importing of the desired package. So thanks for the tip tdelaney.

Python: Module Import Error for the billionth time

2 Answers2