I have a CI/CD pipeline running in Gitlab. The first few lines of the pipeline configuration file .gitlab-ci.yml looks like this:
# jobs can use following stages
stages:
- lint
- build
- test
- deliver
- publish
image: python:3.7
This statement tells me that all CI/CD tests are going to be run in a container based on the python:3.7 image.
My software comprises a set of data processing modules, and each one comes with a set of tests in using pytest. The challenge is that the libraries used by each data processing module can vary widely, and can change often. This means that each time somebody submits a new version of a data processing module which requires new libraries, we have to edit the .gitlab-ci.yml file to add statements like the 3rd line (with pip install):
script:
- python3 -m pip install --user --upgrade pip setuptools twine
- python3 -m pytest -sv --cov=. --cov-report=html tests/
Over time, more pip install lines would have to be edited or added every time a new library is required, or a new library version is required.
Needless to say, this increases the burden on DevOps to ensure that the CI/CD configuration is up-to-date. In addition, it makes it difficult to figure out the configuration required for the production container which bundles all of the data processing software.
One solution I imagine is that each data processing module would have to be tested in a Docker container designed by the module designer, and the image name would be, say, DEVELOPERS_DOCKER_IMAGE. Then the pytest command in the Gitlab CI/CD script would be something like:
script:
- docker run --name=pytest_container -it --rm DEVELOPERS_DOCKER_IMAGE python3 -m pytest -sv --cov=. --cov-report=html tests/
(with additional Docker mounts to store code coverage results).
However, this would require spinning up a Docker container (in this case based on the Docker image DEVELOPERS_DOCKER_IMAGE), from within another Docker container (in this case the CI/CD container based on the python:3.7 image).
Questions:
- Is this possible?
- Is this method recommended or discouraged?
- Are there other solutions which are either better or more in line with Docker idioms and/or best practices?
I appreciate any help I can get.
