0

I am hosting my project on gitlab. I have the CI/CD, which is scheduled to run every 24h to run a certain python script. That script generates a csv file, which is stored in an artifact. I need to access the csv file in other python files of the project, which then parse the csv into dataframes and so on. I managed to run the python script in the pipeline and to generate the artifact, but I don't know how to access it into another python script.

Could I use the artifact as a parameter in the script or how best could I handle this? This is my .gitlab-ci.yml file:

stages:          # List of stages for jobs, and their order of execution
  - build
  - test
  - deploy


build-job:
  stage: build
  script:
    - echo "Running python script to generate csv artifact..."
    - python to_del.py
  artifacts:
    paths:
      - test.csv

expose-job:
  stage: test
  script:
    - echo "Exposing the artifact..."
  artifacts:
    expose_as: 'artifact_file_txt'
    paths: ['test.csv']


deploy-job:      
  stage: deploy  
  script:
    - echo "Run update of csv files based on artifacts??"

In the build-job I am running the python script which generates the artifact, then I am exposing it in the expose-job and I would need to save it to a certain location in the gitlab project in the deploy-job. Is this possible?

martineau
  • 119,623
  • 25
  • 170
  • 301
kitten_world
  • 65
  • 10
  • This is less of a Python issue but more a CI/CD issue with gitlab, [this thread](https://stackoverflow.com/questions/68179565/gitlab-ci-cd-pass-artifacts-variables-between-pipelines) may have what you are looking for. Tags updated to ones more appropriate and relevant for your issue. – metatoaster Apr 21 '22 at 08:18

1 Answers1

0

You can download the artifact using this url.

https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/download?job=<job_name>

You can use the CI_VARIABLE CI_COMMIT_REF_NAME in this URL.

However, if your script exploiting the file is executed at another step, you won’t need to do this, you just have to add the following parameter to you consuming job.

  dependencies:
    - expose-job

If I understand well, you would like your deploy-job to add the artifact to the repository itself.

Technically you can do it, doing some git magic in the pipeline. But personally, I would not recommend doing the same, which I consider a bad practice.

You may consider adding in the readme a magic link to the artifact for instance, or just rely on the artifacts downloading in your consuming script.

Floh
  • 745
  • 3
  • 16
  • Thanks! If I download the artifact using the given url, inside the consuming script, the repository won't be updated with the csv file, right? How can I make sure the csv file is downloaded and available at the start of the consumable script? I am new to CI/CD and don't know how to handle it properly... – kitten_world Apr 21 '22 at 08:58
  • 0 No it won't. Probably the best is to download it with the requests lib. – Floh Apr 21 '22 at 10:13
  • You can commit things to the repository in a job script. – sidney Apr 22 '22 at 11:47