Dependency management between multiple Python modules

Question

I have a project with multiple Python modules, each of which has its own virtual environment.

Project Structure:

data-reader (Python module)
    data_reader
        reader.py
    venv (virtual environment directory for data-reader)
    requirements.txt
data-writer (Python module)
    data_writer
        writer.py
    venv (virtual environment directory for data-writer)
    requirements.txt
commons (Python module)
    commons
        utils.py
    venv (virtual environment directory for commons)
    requirements.txt
    setup.py

I want to install commons as a dependency in both modules: data-reader and data-writer and preferably as an editable project dependency.

I created setup.py in commons and added the following requirement to the requirements.txt in both modules:

-e commons

When I activate the virtual environment for one of the modules and install its requirements I can run scripts from commons in the Python interpreter using the terminal, which is the expected and desired outcome, but in the Intellij IDEA IDE, I get an error underlining the import statement and with this error message: Unresolved reference 'commons'

from commons import utils

I don't know if it is a problem in IDEA or in the approach I am using.

What is the recommended way to add and manage such dependencies?
Is adding this dependency as an editable project one a good idea in the first place or there is other recommended approaches?

your approach is good, there is no serious problem in it. When your product codebase grows, you always end up with something similar. — Andrii Maletskyi, May 18 '18 at 13:56
I have edited the question to include the error message, which is `Unresolved reference 'commons'` — Mousa, May 18 '18 at 13:57
The solution which will work, but it is hard to maintain: when you change commons, create a sdist release using setup.py, and then in other modules install that release with pip. — Andrii Maletskyi, May 18 '18 at 14:02
actually, it is a huge overkill to use many packages for a small project, written by one man. While project grows, it is split in such packages, and every package is developed independently, and has its own vcs repository — Andrii Maletskyi, May 18 '18 at 14:05
Thank you for you comments. The project is not small and it is of course versioned. Our team is not small either and that's why we care very much about the best practices and we are trying to keep the required manual configuration as minimum as possible for other developers when pulling any changes. — Mousa, May 18 '18 at 14:17
As far as I have seen running `pip` with the option `-e` creates the same source distribution as `sdist` which running it manually didn't help in the IDEA problem — Mousa, May 18 '18 at 14:22
in pycharm in `file -> settings -> Project (project name) -> project interpreter ` did you set it to the proper venv? This is different from activating the venv in a command shell. — e.s., May 18 '18 at 16:43
Yes, every module has the correct interpreter which is his virtual environment — Mousa, May 18 '18 at 17:28

score 0 · Answer 1 · answered May 09 '20 at 16:24

The problem seems to be related with Intellij / Pycharm.

You should be choosing the interpreter used by Pycharm to match the one of the virtualenvironment.

If you don't, Pycharm will be using the default python interpreter where the libraries you are importing are not installed.

You can select it with settings -> preference -> Project Interpreter (in the left menu bar). Then add the local virtualenv to the specific python project, for the three of them.

If it helps, the official documentation is:

https://www.jetbrains.com/help/pycharm/creating-virtual-environment.html

Thanks for you answer. It is indeed related to IntelliJ/PyCharm. Actually, it turned out that it is a very old issue which is not getting fixed properly: https://youtrack.jetbrains.com/issue/PY-976 Unfortunately, your suggested workaround does not also work for me, that is something I am doing anyway to ease up some other processes. — Mousa, May 13 '20 at 07:13

score 0 · Answer 2 · answered May 13 '20 at 07:23

As it has been a while, I am posting an answer for other readers. The approach turned out to be good for us and working fine so far.

The issue is with IntelliJ/PyCharm and it turned out to be a very old one which is still not properly fixed: https://youtrack.jetbrains.com/issue/PY-976. A couple of workarounds are reported in the issue itself. I personally have not tried all of them but none of what I have tried so far worked for me.

Another thread is available here: PyCharm does not recognize modules installed in development mode

Dependency management between multiple Python modules

2 Answers2