4

In our current c-project, we use python scripts for support and testing purposes such as unit testing, integration testing, benchmarking and communication.

Current folder structure (most files not shown):

.
|-- workingcopy1
|-- |-- config
|-- |-- |-- __init__.py
|-- |-- |-- parameters_one.py
|-- |-- |-- parameters_two.py
|-- |-- |-- parameters_three.py
|-- |-- src
|-- |-- |-- main_application_c_code.c
|-- |-- tests
|-- |-- |-- tools
|-- |-- |-- |-- display_communication_activity.py
|-- |-- |-- run_all_unit_tests.py
|-- |-- tools
|-- |-- |-- script1.py
|-- |-- |-- script2.py
|-- |-- |-- script3.py
|-- |-- utils
|-- |-- |-- python_utils
|-- |-- |-- |-- __init__.py
|-- |-- |-- |-- communication_utils.py
|-- |-- |-- |-- conversion_utils.py
|-- |-- |-- |-- constants.py
|-- |-- |-- |-- time_utils.py
|-- workingcopy2
...
|-- workingcopy3
...

Some python files are intented to be executed as script files ($ python script1.py) and some are intended to be included as modules in other python files.

What we would like to achive is a structure that enables us to have parameter and utility functions that can be used by:

  • Test code
  • Other utility codes
  • Smaller python application used for monitoring of our system. I.e. custom benchmarking tools

It should also be possible to have several workingcopies checked out

Up until this date, all scripts have following lines at top:

import os, sys
current_path = os.path.abspath(os.path.dirname(__file__))
sys.path.append(os.path.join(current_path, '..', 'utils')) # Variates depending on script location in file tree
sys.path.append(os.path.join(current_path, '..', 'config')) # Variates depending on script location in file tree
import python_utils.constants as constants
import config.parameters_one as parameters

With about 20+ script files this has become hard to maintain. Is there any better way to achive this?

h_enke
  • 43
  • 7

1 Answers1

3

You should convert your folders into Python modules by adding an empty __init__.py file into it.

Also, you can add the Python shebang, so they are executable without explicitly calling the Python command from shell (Should I put #! (shebang) in Python scripts, and what form should it take?).

Once your folders are modules, you have to add only the main source path and you will be able to import the children modules in an easier manner. Furthermore, you should use a virtual environment (virtualenv) that can handle the paths for you (http://docs.python-guide.org/en/latest/dev/virtualenvs/) (and maybe virtualenvwrapper that allows you extra functionality)

I wanted to add a couple of additional strategies you could use here: One of the cool things about python is that everything is an object so you could import and pass your script modules as a variable to a function that run's them, initialising the appropriate path. Also, the function could "discover" the scripts by looking into the folder and walking through it.

Again, all this can be easily handled from pre-post activate virtualenvwrapper hooks.

Community
  • 1
  • 1
zom-pro
  • 1,571
  • 2
  • 16
  • 32
  • I am trying but I cannot understand from this answer how virtualenvwrapper fixes the problem of having to manually append the utils and config folders with sys.path.append. I think __init__ may solve that problem if the `__init__.py` are not blank. An example on how the problem is solved would be much appreciated. – ferreiradev Nov 30 '22 at 03:14
  • @ferreiradev things have moved on these days. I haven't seen virtualwrapper used in years and most of the modern development is done using poetry (or pip and venv). Check what poetry-demo creates as an structure for a python project as shown here https://python-poetry.org/docs/basic-usage/ – zom-pro Dec 11 '22 at 19:27