How can I pass GitLab artifacts to another stage?

Question

I'd like to use GitLab CI with the .gitlab-ci.yml file to run different stages with separate scripts. The first stage produces a tool that must be used in a later stage to perform tests. I've declared the generated tool as an artifact.

Now how can I execute that tool in a later stage job? What is the correct path, and what files will there be around it?

For example the first stage builds artifacts/bin/TestTool/TestTool.exe and that directory contains other required files (DLLs and others). My .gitlab-ci.yml file looks like this:

releasebuild:
  script:
    - chcp 65001
    - build.cmd
  stage: build
  artifacts:
    paths:
      - artifacts/bin/TestTool/

systemtests:
  script:
    - chcp 65001
    - WHAT TO WRITE HERE?
  stage: test

The build and test jobs run on Windows if that's relevant.

score 223 · Answer 1 · edited Mar 31 '23 at 20:49

223

Use dependencies. With this config, the test stage will download the untracked files that were created during the build stage:

build:
  stage: build
  artifacts:
    untracked: true
  script:
    - ./Build.ps1

test:
  stage: test
  dependencies:
    - build
  script:
    - ./Test.ps1

edited Mar 31 '23 at 20:49

Josh Correia

3,807
3
33
50

answered Jul 02 '16 at 02:09

user1495793

2,305
1
12
5

I tried adding dependencies with both stageName(build) or Jobname(build). happens to be same in your example. but that did not work.There is a lot of discussion around using cache/artifacts as well but no luck yet. @peter – ravikanth Aug 04 '16 at 13:20
15

Finally got it to work! The key point here is dependencies should be used along with artifacts. Only the artifacts that were included would be available for consumption in subsequent stage. Needless to say, be conservative on what is being uploaded. I would say use expire_in. Otherwise we could end up wasting lot of storage. These artifacts are uploaded to gitlab in the build job/stage/step and downloaded in test. – ravikanth Aug 04 '16 at 16:00
69

Do you really have to use dependencies? Gitlab documentation states `Note that artifacts from all previous stages are passed by default.`. The question is when do you need to use dependencies. – Aug 02 '18 at 14:56
7

The documentation clears this up quite well: https://docs.gitlab.com/ee/ci/yaml/#dependencies – chetbox Sep 27 '18 at 08:39
10

@Josef artifacts from all previous **stages** are passed by default (Not from the previous jobs) – Vivek Nov 11 '18 at 10:33
12

the documentation is confusing - what does it mean that artifacts of "all previous stages" are being passed? If I understand it correctly, if all previous stages are always passed, then all the data from all the jobs will be available and I do not need to use the "dependencies" tag. The only case I can think of is, if I DO NOT want that behavior and only pass artifacts from one or a selected few jobs. – skofgar Apr 25 '19 at 16:42
3

@Josef when you don't need all artifacts from all previous stages for current job. Let's say you had 10 GB of binaries generated by build stage, but your final stage just sends some emails about successful build - you don't need to download all 10 GB for this job – Ezh May 27 '19 at 12:22

Chuan · Answer 2 · 2019-08-02T21:36:24.273

217

Since artifacts from all previous stages are passed by default, we just need to define stages in correct order. Please try the example below, which could help understanding.

image: ubuntu:18.04

stages:
  - build_stage
  - test_stage
  - deploy_stage

build:
  stage: build_stage
  script:
    - echo "building..." >> ./build_result.txt
  artifacts:
    paths:
    - build_result.txt
    expire_in: 1 week

unit_test:
  stage: test_stage
  script:
    - ls
    - cat build_result.txt
    - cp build_result.txt unittest_result.txt
    - echo "unit testing..." >> ./unittest_result.txt
  artifacts:
    paths:
    - unittest_result.txt
    expire_in: 1 week

integration_test:
  stage: test_stage
  script:
    - ls
    - cat build_result.txt
    - cp build_result.txt integration_test_result.txt
    - echo "integration testing..." >> ./integration_test_result.txt
  artifacts:
    paths:
    - integration_test_result.txt
    expire_in: 1 week

deploy:
  stage: deploy_stage
  script:
    - ls
    - cat build_result.txt
    - cat unittest_result.txt
    - cat integration_test_result.txt

And in case to pass artifacts between jobs in different stages, we can use dependencies together with artifacts to pass the artifacts, as described from the document.

And one more simpler example:

image: ubuntu:18.04

build:
  stage: build
  script:
    - echo "building..." >> ./result.txt
  artifacts:
    paths:
    - result.txt
    expire_in: 1 week

unit_test:
  stage: test
  script:
    - ls
    - cat result.txt
    - echo "unit testing..." >> ./result.txt
  artifacts:
    paths:
    - result.txt
    expire_in: 1 week

deploy:
  stage: deploy
  script:
    - ls
    - cat result.txt

edited Aug 02 '19 at 21:36

answered Jun 11 '19 at 19:53

Chuan

3,103
1
19
23

5

Very clear explanation, thank you. If a stage names an artifact by the same name as an artifact from a previous stage, does the original artifact get overwritten? – Michael Osofsky Aug 02 '19 at 18:45
3

@MichaelOsofsky You can name the artifact by the same name, the original artifact will not get overwritten by the one from next stage with the same name. The next stage only downloads the artifact from the former stage, it is a copy of it. I name them differently in the example mainly due to unit test and integration will be executed in parallel. If we remove .e.g integration test job, all jobs will execute in sequence, then we can use the same name for all the artifacts without any confusion. FYI, I update the answer with one more example. – Chuan Aug 02 '19 at 21:49
In your example I see you're appending to result.txt. If you overwrote result.txt in the job unit_test, I assume the job deploy would never have access to the contents of result.txt from the job build. I'm just asking to ensure I never cause this type of bug in my scripts. – Michael Osofsky Aug 02 '19 at 21:52
5

According to the log, the deploy stage will download both result.txt from build and test stages, but the later one will overwrite the former one. – Chuan Aug 02 '19 at 22:03
4

BTW, the original artifact is not touched and always available for downloading from CI/CD -> Pipelines, then click the dropdown button for Artifacts on the right, you will find all the artifacts of all stages. – Chuan Aug 02 '19 at 22:09
Note you have to declare files as artifacts even if you don't need them in Gitlab artifact storage, because otherwise they won't be transferred to the next stage. – Putnik May 03 '20 at 14:22
Just for demo purpose. @Putnik – Chuan May 04 '20 at 11:28
@Chuan How can we send the artifacts(builds) via email on successfully operation – Ravindra Kushwaha Nov 04 '20 at 10:42
so basically, if you define it as `artifacts` it gets passed to next stage, if you don't then nothing gets passed? – cryanbhu Jan 21 '21 at 02:58

k1eran · Answer 3 · 2023-06-27T08:10:13.477

If you want foo/ to be available in the next stage AND it is in your .gitignore you'll need to list it in the artifacts of the stage that creates it, or as explained at here use untracked: true. As I wanted just a subset I didn’t use untracked: true.

The following approach worked for me (with NO dependencies specified in the following stage)

   artifacts:
     paths:
       - foo/
     expire_in: 1 hour

BTW regarding the : expire_in: 1 hour part:
I read at https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2133 there's no way to get artifacts to automatically expire at the conclusion of pipeline and the default retention was surprisingly long (30 days by default) - hence the time-based kludge to get rid of them - see https://docs.gitlab.com/ee/ci/yaml/

➕1️⃣ for mentioning the expiration info – olfek Jan 15 '23 at 10:58 — olfek, Jan 15 '23 at 10:58

score 0 · Answer 4 · answered Jul 20 '23 at 19:35

If you use the needs: keyword, the default artifact management behavior changes. The only artifacts available are from "upstream" jobs on the needs graph. Also, the dependencies: keyword cannot be used with the needs: keyword.

To address the pipeline snippet from the question, adding a needs relationship to the job that creates the required artifacts is all that is needed:

releasebuild:
  script:
    - chcp 65001
    - build.cmd
  stage: build
  artifacts:
    paths:
      - artifacts/bin/TestTool/

systemtests:
  script:
    - chcp 65001
  stage: test  
  needs:
    - job: releasebuild
      artifacts: true

NOTE: The needs:artifacts: keyword defaults to true and can be omitted. When set to false, the job won't load the upstream artifacts.

score 0 · Answer 5 · answered Aug 31 '23 at 05:00

You can use needs section. Docs needs.

releasebuild:
  needs: [] # Pass there dependencies that needs to start this job or leave it empty
  ...
  artifacts:
    paths:
      - artifacts/bin/TestTool/

systemtests:
  needs: ["releasebuild"]
  script:
    # Now you can do whatever you want with artifacts/bin/TestTool/ here
    # Example:
    - cp artifacts/bin/TestTool/* test/*
  ...

The same for needs:artifacts, just another syntax

releasebuild:
  needs: [] # Pass there dependencies that needs to start this job or leave it empty
  ...
  artifacts:
    paths:
      - artifacts/bin/TestTool/

systemtests:
  needs:
    - job: releasebuild
      artifacts: true
  script:
    # Now you can do whatever you want with artifacts/bin/TestTool/ here
    # Example:
    - cp artifacts/bin/TestTool/* test/*

This is pretty same that dependencies ones.

How can I pass GitLab artifacts to another stage?

5 Answers5

Linked