Python: Structuring a project with utility functions shared across modules at different levels

Question

I have python 3.10 project that uses a combination of scraping websites, data analysis, and additional APIs. Some utility modules may be used by the scraping and data analysis modules. I'm fundamentally misunderstanding something about how imports work in Python. For example, in sl_networking.py, I try to import the Result class from result.py:

from ...util.result import Result

Producing the error:

PS C:\Development\TradeAssist> & c:/Development/TradeAssist/.venv/Scripts/python.exe c:/Development/TradeAssist/libs/scrapers/sl/sl_networking.py
Traceback (most recent call last):
  File "c:\Development\TradeAssist\libs\scrapers\sl\sl_networking.py", line 1, in <module>
    from ...util.result import Result
ImportError: attempted relative import with no known parent package

The project structure I'm currently using is:

TradeAssist
|__libs
|  |__broker_apis
|  |  |__ibapi
|  |__data_analysis
|  |  |__sl
|  |  |__ta
|  |__scrapers
|  |  |__sl
|  |  |  |  sl_auth.py
|  |  |  |__sl_networking.py
|  |  |__ta
|  |__util
|     |__result.py
|__tests
   |__test_sl.py
   |__test_ta.py

If I have a common utility function that I expect to use within the data_analysis and scraper modules, how should I be structuring my project and handling imports?

Answer 1

This is not a Python question, but rather a VSCode question. The way VSCode runs Python files by default is a little bit dumb. When you click the little triangle to "Run Python File" on test_sl, VSCode will run a command such as this:

/usr/local/bin/python3 /path/to/TradeAssist/tests/test_ta.py

Python by default initialises sys.path to contain the directory containing the file being run, in addition to its own libraries, and whatever is in your PYTHONPATH variable. This means your sys.path looks something like this (with ... being Python's libraries)

['/path/to/TradeAssist/tests', ...]

However, this means code in lib is not reachable; only files inside tests will be correctly found by Python. I.e. out of the box, VSCode is only able to run Python files that are in your root source folder (e.g. src, or directly in the root workspace folder).

There are several solutions.

The first one is simplest: decide that your TradeAssist is the source root, and put any files you want to execute directly there. They will be able to import any files under them, and the files under them will be able to use both relative and absolute imports correctly. This has obvious disadvantages — you would be self-limiting yourself to one directory.

The second is to set up VSCode to tell Python where your source root(s) are, by defining PYTHONPATH. This is fairly complex.

The third is the simplest, and likely the correct case here: use VSCode's testing functionality. Configure the testing framework (presumably using unittest and test_*.py test file pattern), then just run the tests. If you consider TradeAssist to be your source root, it will work correctly (i.e. import libs.scrapers.sl.sl_networking will work correctly inside your tests, and from ...util.result import Result will work correctly inside your sl_networking.py). The testing function will take care of running the tests for you in a correct fashion.

However, if you want to consider libs to be your source root (i.e. you would like to do import scrapers.sl.sl_networking), or if you want to be able to run arbitrary files, not just tests, then you have to fall back to method #2: messing with PYTHONPATH.

tl;dr: Don't run test files manually, let VSCode do it for you by setting up tests correctly.

Answer 2

Relative imports only work when the code is executed from the outermost parent root. In the current scenario, you can only execute the code at or above libs directory.

python -m scrapers.sl.sl_networking

should work fine if you are running this at libs directory.

Once the project is structured, it is easy to run the individual scripts from the top parent directory using -m flag, as no refactoring will be required. If the code has to be executed from the script parent directory, the following has to be done:

1. Use absolute imports instead of relative imports.
2. Add the directory to the path python searches for imports. This can be done in several ways. Add it to the PYTHONPATH env variable or use any of the sys.path.append or sys.path.insert hacks, which can be easily found.