2

I have a problem similar to that described here How to fix "ImportError: No module named ..." error in Python? but I cannot fix it with the suggestion to set PYTHONPATH.

my directory looks like:

- project
    - python
        - src
          - ml
            - __init__.py
            - classifier_dnn.py
            - util.py           
            - vectorizer
              - fv_davison.py
              - __init__.py

And I am running classifier_dnn.py at the project folder path:

~project&PYTHONPATH=/home/project/
~project$python3 /home/project/python/src/ml/classifier_dnn.py /home/project/data/labeled_data_all.csv /home/project/output

But an error is generated when classifier_dn imports ml.util:

Traceback (most recent call last):
  File "/home/project/chase/python/src/ml/classifier_dnn.py", line 5, in <module>
    from ml import util
ImportError: No module named 'ml'

I have also tried setting PYTHONPATH=/home/project/python or PYTHONPATH=/home/project/src but the same error happens.

When I test this in PyCharm, it works if set python/src to become source root, regardless what working directory is. But I cannot figure out how to set this properly when I run this from command line.

Any help please

Thanks

Ziqi
  • 2,445
  • 5
  • 38
  • 65
  • Both your `PYTHONPATH` variants are wrong or is it just a typo? Guessing from your directory tree it should be `PYTHONPATH=/home/project/python/src`. Note, that this is a dubious way to build/use a library. – mkiever Jul 17 '17 at 16:37

2 Answers2

3

I have a writeup on this but I'll copy the relevant text inline.

You have two problems:

  • You need to export PYTHONPATH (export PYTHONPATH=/full/path/to/src)
  • You should run with python -m module.path instead of python path/to/file.py

The core problem is python path/to/file.py puts path/to on the beginning of the PYTHONPATH (sys.path)

This causes imports to start from (in your case) src/ml instead of the expected src.

By using -m, you avoid this path munging and it will correctly keep src as the beginning of your sys.path.

When I test this in PyCharm, it works if set python/src to become source root, regardless what working directory is. But I cannot figure out how to set this properly when I run this from command line.

You can do this by cding into the src directory

anthony sottile
  • 61,815
  • 15
  • 148
  • 207
  • Technically correct answer. Yet, I see bigger problems in the idea to have an executable script inside a library. Except maybe for testing. – mkiever Jul 17 '17 at 16:47
  • It's not a problem if you guard your main with `if __name__ == '__main__'` (as you should be doing anyway!) – anthony sottile Jul 17 '17 at 16:52
  • I was not only thinking about this problem, more about this: How do you execute your script once the library is installed somewhere? You don't want to enter/know the complete library path every time you use an executable (and don't forget about `-m`). – mkiever Jul 17 '17 at 17:02
  • You use setuptools `console_scripts`, for example: [the console_scripts section for pre-commit](https://github.com/pre-commit/pre-commit/blob/v0.15.2/setup.py#L52-L54) – anthony sottile Jul 17 '17 at 17:13
  • thanks this works a treat! Yes I am running the code for experimentation purposes and having no intention to package it as a real piece of software – Ziqi Jul 17 '17 at 19:08
  • @Anthony Sottile (have been catching up, only used distutils until now) I am a bit surprised setuptools is doing it this way. I don't like it. Nonetheless, thanks for the hints. – mkiever Jul 18 '17 at 10:16
  • fwiw `console_scripts` is the new best practice suggested by pypa – anthony sottile Jul 18 '17 at 16:37
-1

You should not put executable scripts into a library structure.

Only library modules should go into the package structure. Executables should stay outside it. This is important for installation and day-to-day use. Once you have installed your package (using distutils for example) using it would be complicated, if you leave the executable in the library. You would have to give the complete installation path into the python library (and don't forget about -m!) every time or create shortcuts or other workarounds.

Seperate executables and library. Executables are installed to something like usr/local/bin and are callable without path, -m or other problems, and your library is installed into the python library.

mkiever
  • 894
  • 5
  • 13