How to stick with a Gitlab CI Runner within a pipeline?

Question

In our Gitlab project group, we are using multiple shared runners for CI. However, some of the jobs have dependencies, such that the previous job must have been executed on the same runner.

Here is an Example:

Job 1 builds a docker container
Job 2 checks the docker for execution, so it needs the docker image from job 1
Job 3 pushes the docker image to a container hub, so it needs the docker image from job 1

Now, with multiple shared runners it may happen, that job1 is executed on runner 1 and job 2 and 3 on a different runner than runner 1. This throws an error in job 2 and 3 as the docker image is locally not available on that runner.

On the other hand, we need multiple runners due to the amount of computation in our project. So it would be great if once a runner is picked in a specific job, it keeps the same runner for the ongoing jobs.

Any ideas to solve this?

I just answered a similar question over here: https://stackoverflow.com/questions/65902682/how-to-run-gitlab-ci-jobs-in-the-same-instance/65904658#65904658. Does that help? — Adam Marshall, Jan 26 '21 at 15:57

Thomas B · Answer 1 · 2021-01-26T16:13:47.013

2

Gitlab scheduling doesn't permit it easily. The balancing between runners work as the following:

When a job is created on Gitlab instance, it registers a job in pending state
Runners check every 3 seconds (3s by default, configured with check_interval) if there are jobs in the pending queue. If yes, and if the runner is the right runner for the job (for example if job tags are compliant with runner), then the runner launch N jobs from the queues, N limited by the maximum number of concurrent job per runner (concurrent option)

So this isn't Gitlab itself that schedule which runner runs which job. Gitlab just put jobs to run in a queue, runners frequently check the queue and select jobs. It's great for scalability but not great for your use case.

In my point of view, you have 2 options:

First, put a specific tag on only one runner and use it on jobs that need to be run on the same host
Second, more flexible, is to store the resulting docker image in your Gitlab project registry at the end of the build and pull it from any job that needs the image (job2 and job3) See https://docs.gitlab.com/ee/user/packages/container_registry/#build-and-push-by-using-gitlab-cicd

edited Jan 26 '21 at 16:13

answered Jan 26 '21 at 15:40

Thomas B

63
7

1

Thanks for your answer and your recommendations! Honestly, this is what I wanted to avoid: I only want to upload docker images that pass the execution checks. And for the first option we would limit ourselves to one runner, which is unfeasible at the current size of our project. – maxgemilian Jan 26 '21 at 20:04
Looking through the documentation for Gitlab Runner and couldn't find the information you provided, so is it possible to present a link for the information your shared? – OMG-1 Feb 03 '21 at 12:33
@OMG-1 I don't found the dedicated page in the documentation anymore but here are an issue asking for load balancing (which isn't currently possible because of the scheduling mechanism): [Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/15963). [Comment of Gitlab CEO](https://gitlab.com/gitlab-org/gitlab/-/issues/15963#note_214874016), [comment about a workaround that describe the same runner behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/15963#note_401418264) – Thomas B Feb 04 '21 at 11:23

How to stick with a Gitlab CI Runner within a pipeline?

1 Answers1