Assign Gitlab Runner daemon's pod and the jobs's pods to two separate node groups in Kubernetes when using Kubernetes executor

Question

We're using Gitlab Runner with Kubernetes executor and we were thinking about what I think is currently not possible. We want to assign the Gitlab Runner daemon's pod to a specific node group's worker with instance type X and the jobs' pods to a different node group Y worker nodes as these usually require more computation resources than the Gitlab Runner's pod.

This comes in order to save costs, as the node where the Gitlab runner main daemon will always be running, then we want it to be running on a cheap instance, and later the jobs which need more computation capacity then they can run on different instances with different type and which will be started by the Cluster Autoscaler and later destroyed when no jobs are present.

I made an investigation about this feature, and the available way to assign the pods to specific nodes is to use the node selector or node affinity, but the rules included in these two configuration sections are applied to all the pods of the Gitlab Runner, the main pod and the jobs pods. The proposal is to make it possible to apply two separate configurations, one for the Gitlab Runner's pod and one for the jobs' pods.

The current existing config consists of the node selector and nodes/pods affinity, but as I mentioned these apply globally to all the pods and not to specified ones as we want in our case.

Gitlab Runner Kubernetes Executor Config: https://docs.gitlab.com/runner/executors/kubernetes.html

doc you have config could you please share for ref or share doc of github repo for yaml ref it's little confusing. looks like you have single config for main pod and jobs pod right ? — Harsh Manvar, Jul 22 '22 at 09:51
Yes, exactly, the config is the same for the main GitLab runner pod and the jobs' pod. I adapted the question description and included useful config links — Rshad Zhran, Jul 22 '22 at 09:52
what you have mentioned above is possible indeed `Gitlab Runner daemon's pod to a specific node group's worker with instance type X and the jobs' pods to a different node group Y worker` however in this case two separate config needs to be there. — Harsh Manvar, Jul 22 '22 at 09:53

score 4 · Accepted Answer · answered Aug 23 '22 at 16:07

This problem is solved! After a further investigation I found that Gitlab Runner's Helm chart provide 2 nodeSelector features, to exactly do what I was looking for, 1 for the main pod which represents the Gitlab Runner pod and the other one for the Gitlab Runner's jobs pods. Below I show a sample of the Helm chart in which I set beside each nodeSelector its domain and the pod that it affects.

Note that the first level nodeSelector is the one that affects the main Gitlab Runner pod, and the runners.kubernetes.node_selector is the one that affects the Gitlab Runner's jobs pods.

gitlabUrl: https://gitlab.com/
...
nodeSelector:
  gitlab-runner-label-example: label-values-example-0
...

runnerRegistrationToken: ****
...
runners:
  config: 
    [[runners]]
        name = "gitlabRunnerExample"
        executor = "kubernetes"
        environment = ["FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true"]
        
        [runners.kubernetes]
            ...
        [runners.kubernetes.node_selector]
            "gitlab-runner-label-example" = "label-values-example-1"

        [runners.cache]
            ...
            [runners.cache.s3]
                ...
...

score 1 · Answer 2 · answered Jul 22 '22 at 14:19

1

using the helm chart, there is an additional configuration part where you can specify, well, additional configuration

one of the them is especially the node selector for jobs pods and another one for toleration.

The combination of that and some namespace level config should allow you to run the 2 kinds of pod on different node types

answered Jul 22 '22 at 14:19

Jean-Pascal J.

146
1
8

I already make use of the node selector, but as I mentioned above, this configuration is applied globally to all the pods, and there's no possibility to separate it. – Rshad Zhran Jul 22 '22 at 20:36
1

I upvoted your answer however it's not completely true, but yes the first line and it's not that clear yet. The tolerations have nothing to do with this case, I mean they don't help to refer one pod or another as I was looking for. I added a detailed answer with a sample chart that solved the problem. https://stackoverflow.com/a/73461907/7611357. Thanks anyway! – Rshad Zhran Aug 23 '22 at 16:08

Assign Gitlab Runner daemon's pod and the jobs's pods to two separate node groups in Kubernetes when using Kubernetes executor

2 Answers2