How to run the Gitlab CI jobs with different runners independent from each other?

Question

Description

As I have found from this Stackoverflow question and Gitlab-CI official documentation, by using tags keyword, different runners can be triggered for a single project. So I have registered different runners on my servers (one runner for each (staging and production) server) with dashboard_staging and dashboard_production tag names.

Everything works well and in order to start gitlab-runner properly, I have executed the following commands:

sudo gitlab-runner verify  # Everything was ok
sudo gitlab-runner start  # It was started successfully on both servers

Then I have committed the changes and push them on Gitlab and it was triggered successfully.

Problems

I have executed the above commands, but one of the pipelines is still pending for a runner.

Since build stage has not been done completely, it wouldn't progressed for the tag in which its job has been done.

Code

stages:
  - test
  - build
  - deploy

cache:
  untracked: true
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - .yarn

dashboard:test:staging:
  stage: test
  tags:
    - dashboard_staging
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:test:production:
  stage: test
  tags:
    - dashboard_production
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:build:staging:
  stage: build
  tags:
    - dashboard_staging
  only:
    - staging
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"

dashboard:build:production:
  stage: build
  tags:
    - dashboard_production
  only:
    - staging
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"


dashboard:deploy:staging:
  stage: deploy
  tags:
    - dashboard_staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"


dashboard:deploy:production:
  stage: deploy
  tags:
    - dashboard_production
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"

Questions

How can I resolve this pending issue in Deploy stage?

Is there any way to run the defined tags independently from each other?

score 1 · Answer 1 · answered Sep 24 '21 at 01:52

If I'm understanding what you'd like correctly, you want the "staging" jobs to run independently of the "production" jobs, and ignore the fact that the other job in the same stage may not have completed yet.

This is what the needs keyword is for (reference). Since you have not defined any needs keyword on your jobs, each job will wait for the whole previous stage to complete before it kicks off. The default behavior of GitLab CI is to run all jobs within a stage in parallel, then each stage in series. You can then override that with the needs keyword to start jobs independent of the stage they are in.

Try the following:

stages:
  - test
  - build
  - deploy

cache:
  untracked: true
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - .yarn

dashboard:test:staging:
  stage: test
  tags:
    - dashboard_staging
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:test:production:
  stage: test
  tags:
    - dashboard_production
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:build:staging:
  stage: build
  tags:
    - dashboard_staging
  only:
    - staging
  needs: ["dashboard:test:staging"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"

dashboard:build:production:
  stage: build
  tags:
    - dashboard_production
  only:
    - staging
  needs: ["dashboard:test:production"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"


dashboard:deploy:staging:
  stage: deploy
  tags:
    - dashboard_staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:staging"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"


dashboard:deploy:production:
  stage: deploy
  tags:
    - dashboard_production
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:production"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"

Do you have any idea why one of my runners is not working properly? All of the steps are as the same. — Mostafa Ghadimi, Sep 24 '21 at 04:08
Do you have more information other than "is not working properly"? What's error messages are you seeing? It should be picking up jobs based on your screenshot showing green status of both runners, and the tags matching what you have in your CI. I don't know what type of runners you're trying to use either - are they docker runners? Shell runners? — Patrick, Sep 24 '21 at 04:18
Shell runners.There isn’t any error but it’s not been triggered. — Mostafa Ghadimi, Sep 24 '21 at 04:34
Are the jobs just sitting in pending then and listed as "stuck" when you click on them? If so, you'll want to read through the logs being output from the runner. If you can post what the runner logs look like, I can probably help more. It's unrelated to this question though - the issue will be in your runner, not your CI script. — Patrick, Sep 24 '21 at 14:08
I see, so it wasn't an issue with the stage completing, it was an issue where one of the jobs _never_ picked up. Assuming it's listed as being "stuck", one of the runners is not configured properly. If you can post runner logs, then I can help troubleshoot more. — Patrick, Sep 25 '21 at 00:13
Thank you Patrick for your descriptive answer, I have also posted my own solution and answers to the questions. — Mostafa Ghadimi, Oct 06 '21 at 04:31

score 1 · Accepted Answer · answered Oct 06 '21 at 03:03

I will post the details about how my problem and question was solved for future readers.

Question 1: How can I resolve this pending issue in Deploy stage?

Answer 1:

This problem was solved after executing the following three commands:

sudo gitlab-runner verify
sudo gitlab-runner start
sudo gitlab-runner run

P.S. If you are non-root and want to execute gitlab-runner without sudo (with your user privilege, you can easily add gitlab-runner to the sudoers by usermod -aG sudo gitlab-runner command.)

For more details, you can visit the following links:

Question 2: Is there any way to run the defined tags independently from each other?

Answer 2: As @Patrick has mentioned, there is a tag existing in gitlab-ci yaml file named needs that exactly works as expected.

stages:
  - test
  - build
  - deploy

cache:
  untracked: true
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - .yarn

dashboard:test:staging:
  stage: test
  tags:
    - dashboard_staging
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:test:production:
  stage: test
  tags:
    - dashboard_production
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:build:staging:
  stage: build
  tags:
    - dashboard_staging
  only:
    - staging
  needs: ["dashboard:test:staging"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"

dashboard:build:production:
  stage: build
  tags:
    - dashboard_production
  only:
    - staging
  needs: ["dashboard:test:production"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"


dashboard:deploy:staging:
  stage: deploy
  tags:
    - dashboard_staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:staging"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"


dashboard:deploy:production:
  stage: deploy
  tags:
    - dashboard_production
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:production"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"

For more information you can read the official documentation of Gitlab CI.

How to run the Gitlab CI jobs with different runners independent from each other?

2 Answers2