Python pytest cannot find modules inside src directory

Question

I'm facing a challenge with my Python project setup that involves module imports, testing using pytest, and working with Jupyter Notebook. My project directory structure looks like this:

project/
├── Dockerfile
├── requirements.txt
├── pyproject.toml
├── jupyter_notebook_config.py
├── notebooks/
│   ├── src/
│   │   ├── my_module.py
│   │   └── module_name.c
│   ├── tests/
│   │   ├── test_my_module.py
│   │   └── test_sub_module.py
│   └── my_notebook.ipynb
└── ...

Inside the pyproject.toml file, I've configured the Python paths for pytest like this using the pytest-pythonpath plugin:

# pyproject.toml
[tool.pytest.ini_options]
python_paths = ["src"]

My Docker container is based on the python:3.11 image and uses a Dockerfile similar to the following:

FROM python:3.11

# Copy the requirements.txt file to the container
COPY requirements.txt /app/requirements.txt

# Install the dependencies from the requirements.txt file
RUN pip install -r /app/requirements.txt

# Copy the pyproject.toml file to the container
COPY pyproject.toml /app/pyproject.toml

# Set working directory
WORKDIR /app

# Run pip install -e . to install pyproject.toml
RUN pip install -e .

# Expose Jupyter Notebook port
EXPOSE 8888

# Copy the custom Jupyter configuration file
COPY jupyter_notebook_config.py /root/.jupyter/

# Run Jupyter Notebook on container startup
CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]

I've attempted to address module import issues by adding the following code at the beginning of my test files and notebooks:

import os
import sys

sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), 'notebooks/src')))

However, despite these efforts, I'm encountering problems where both pytest and Jupyter Notebook cannot locate the modules inside the src directory. How can I properly configure my project so that both pytest and Jupyter Notebook can access the modules within the src directory for importing, testing, and working with notebooks?

To provide additional context, I've encountered similar issues with module imports and working with compiled C extensions and Python files. I've referred to the following discussions for guidance:

1. Unable to Access pytest Test Features Without Modifying sys.path
2. Importing Compiled C Extensions & Python Files from Folder Dynamically in Jupyter Notebook

Additionally, when attempting to run pytest, I'm encountering an error message like the one below:

============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-6.2.5, py-1.11.0, pluggy-1.2.0
rootdir: /app
plugins: pythonpath-0.7.4, anyio-3.7.1
collected 0 items / 1 error                                                    

==================================== ERRORS ====================================
_______________ ERROR collecting tests/test_my_functions_demo.py _______________
ImportError while importing test module '/app/tests/test_my_functions_demo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_my_functions_demo.py:2: in <module>
    from my_functions_demo import add, subtract
E   ModuleNotFoundError: No module named 'my_functions_demo'
=========================== short test summary info ============================
ERROR tests/test_my_functions_demo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.15s ===============================

To illustrate the issue further:

Working Solution:

import src.module_name

# Calculate lengths and elements for the range from 1 to 15 using Cython
sample_range, lengths, elements = src.module_name.calculate_lengths_cython(1, 15)

Not Working Solution:

import module_name

# Calculate lengths and elements for the range from 1 to 15 using Cython
sample_range, lengths, elements = module_name.calculate_lengths_cython(1, 15)