Running a stage after conditional manual stage in a Gitlab pipeline

Question

I'm trying to run a pipeline where the final stage depends on a previous stage with jobs that are conditional and manual.

I made this example .gitlab-ci.yml to demonstrate the point. I am working with three stages:

stages:
    - test
    - publish
    - create tag

There is one job in the Test stage

# Tests Stage
run tests:
    stage: test
    script:
        - run the tests

Three jobs in the Publish stage, all of which are manual and only exist when certain files have changed

.publish:
    stage: publish
    script:
        - publish x
    rules:
        - changes:
            - $DIR/**/*
          when: manual


# Publish Stage
publish package a:
    variables:
        DIR: a
    extends:
        - .publish

publish package b:
    variables:
        DIR: b
    extends:
        - .publish

publish package c:
    variables:
        DIR: c
    extends:
        - .publish

And finally the Create Tag stage, which I only want to run if one of the publish jobs has completed.

# Create Tag Stage
create tag with all packages:
    stage: create tag
    script:
        - git tag

Usually I can use needs to make the Create Tag job depend on a publishing job. But if, for example, I only make changes in the a/ directory, I will get an error for the following because only "publish package a" exists:

needs:
    - "publish package a"
    - "publish package b"
    - "publish package c"

What I really want is something like

needs:
    - "publish package a" if exists
    - "publish package b" if exists
    - "publish package c" if exists

But there's nothing like this as far as I know. What can I do to run the Create Tags job only when the existing jobs in the Publish stage have completed?

score 2 · Accepted Answer · answered Oct 20 '21 at 14:55

What you're looking for is Optional Needs which was introduced with a Feature Flag in version 13.10, and was promoted (available for use without the feature flag) in 14.0. This allows you to have a job like this:

build:
  stage: build
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

rspec:
  stage: test
  needs:
    - job: build
      optional: true

In this example, the build job only runs if the pipeline is for a Branch (and not a tag, merge request, etc.) and if the branch name is the same as the project's default branch. If this isn't the case, this job isn't added to the pipeline at all, so we need to ensure that our needs for the rspec job is optional.

By adding the optional: true attribute to our needs array, we tell Gitlab that the rspec job needs the build job if it exists.

So, for your example:

stages:
    - test
    - publish
    - create tag

# Tests Stage
run tests:
    stage: test
    script:

.publish:
    stage: publish
    script:
        - publish x
    rules:
        - changes:
            - $DIR/**/*
          when: manual


# Publish Stage
publish package a:
    variables:
        DIR: a
    extends:
        - .publish

publish package b:
    variables:
        DIR: b
    extends:
        - .publish

publish package c:
    variables:
        DIR: c
    extends:
        - .publish

# Create Tag Stage
create tag with all packages:
    stage: create tag
    needs:
      - job: "publish package a"
        optional: true
      - job: "publish package b"
        optional: true
      - job: "publish package c"
        optional: true
    script:
        - git tag

This feature was introduced in Issue 30680.

Thanks for the comment Adam! While this works for a number of publish package X, it would require adding a needs for each new packages and setting optional to be true. We have a growing number of jobs that would extend .publish in the future, is there something more sustainable than this solution so it's dynamic? — jimmyhuang0904, Feb 14 '22 at 23:50

Running a stage after conditional manual stage in a Gitlab pipeline

1 Answers1