Src layout to dispense .src prefix in imports? Activate venv in PyCharm terminal for development installs

Question

I want to understand what is considered the correct minimalist way to use setuptools with a "src/ layout" in a way that dispenses using src. prefix in imports?

I have read most of the PyPA and setuptools documentation (and its many use cases), but I can't understand what is considered the correct way of doing this example.

The below layout reproduces what I want to achieve. I can't understand how to get the second import to work instead of the first across all modules of the mylibrary package:

from src.mylibrary.hello_word import hello_function # <- This works.
from mylibrary.hello_word import hello_function  # <- How to get this working?

hello_function()

Using this directory/file structure:

C:\MyProject
│
│   setup.py
│
└───src
    │
    ├──mylibrary
    │      hello_word.py
    │      module_two.py
    │      __init__.py
    │

When I use development mode install with pip install -e . the egg directory is added to the above tree:

    │ (...)
    │ 
    └──mylibrary.egg-info
           dependency_links.txt
           PKG-INFO
           SOURCES.txt
           top_level.txt

With this setup.py:

from setuptools import setup, find_packages, find_namespace_packages

setup(
    name='mylibrary',
    version='0.1',
    package_dir={'': 'src'},
    # packages=find_namespace_packages(where='src'),  # <- I suppose this isn't the deciding factor.
    packages=find_packages(where='src'),
)

The simple hello_world.py module that I want to dispense having to write src. when importing.

def hello_function():
    print("hello world")

The __init__.py is left empty.

I'm using a venv, to my surprise the egg symlink isn't written to the venv sitepackages but to C:\Users\Name\AppData\Roaming\Python\Python38\site-packages...

Python console indicates mylibrary package is found:

>>> from setuptools import find_packages
>>> find_packages(where='src')
['mylibrary']

Answer 1

Update / Workaround

One can follow the steps in this link to mark a folder as a source root. From the link:

Source roots the Source root icon contain the actual source files and resources. PyCharm uses the source roots as the starting point for resolving imports

This way we don't have to install It or to change the path anymore (although It still feels kind of tricky to me)

1. Pycharm has a pip problem:

Whenever I run pip in the pycharm terminal It runs the system's pip for some reason. You can check that with the which pip command. But if I run python -m pip It does the right thing.

2. It should actually recognizes the imports:

That being said, I really don't know why pycharm can't recognize the imports without the src. prefix. I cloned the `cryptography package from github and It recognizes It normally, the src folder is automatically added to path, but I don't know how Yet.

Either way, your code works as expected without the prefix and Vim (using Pyright) recognizes It flawlessly. Even running It on pycharm actually works with all the import errors.

3. What to do now?

Until they fix It or I find a solution I'll just add the src folder to my path in the interpreter settings manually, since I don't want to install It (but I want to be able to peak and go to definitions).

If I really want to install the package I would do It in the terminal or inside pycharm using python -m pip install -e -v .

Answer 2

The problem described results from having to activate the venv inside PyCharm's terminal.

A description of the scenarios you'll likely encounter follows. (The problem isn't immediately obvious because unlike the terminal, functionalities like debugging, running, etc, integrate the venv in a seamless way.)

It should be noted:

• Using the verbose flag -v while installing in development mode gives clues to what pip and setuptools are trying to do.

• The decisive pip messages are based on write permissions of your site-packages, however you won't have to change any of the default permission if activating your venv on the terminal.

• If you are using 1 venv, there will be 3 different site-packages involved (mind the paths).

The 3 options you are likely to try:

Option 1. Run PyCharm as admin, executing the following from the terminal gives:

C:\MyProject>pip install -v -e .

Non-user install because site-packages writeable
(...)
Creating c:\program files\python38\lib\site-packages\mylibrary.egg-link (link to src)

This installs to site-packages (mind the path) in your base Python installation. Something you likely want to avoid, because it pollutes your base installation.

Option 2. Run PyCharm as user. Without activating venv on the terminal.

C:\MyProject>pip install -v -e .

Defaulting to user installation because normal site-packages is not writeable
(...)
Creating c:\users\name\appdata\roaming\python\python38\site-packages\mylibrary.egg-link (link to src)

This installs to site-packages (mind the path) outside your venv, and outside your Python base installation. Something you likely want to avoid, because PyCharm won't recognize the development installation after it's done.

NOTE: The message in the terminal "(...) site-packages is not writeable" refers to the site-packages in your Python base instalation. But, without explicitly activating the venv, even if you set the permissions to writeable, the development instalation won't write to your venv site-packages.

Option 3. Run PyCharm as user. Activating venv on the terminal.

(MyProject_venv) C:\MyProject>pip install -v -e .

Non-user install because user site-packages disabled
(...)
Creating c:\myproject_venv\lib\site-packages\mylibrary.egg-link (link to src)

Here you did write to site-packages in your venv, which is likely what you want.