0

I'm facing a pretty strange Problem.

First of all my setup:
I got a private Gitlab server which uses Gitlab CI Runners on Kubernetes to build Docker Images. For that purpose I use the Kaniko Image. The Runners are provisioned by Gitlab itself with the built-in Kubernetes management. All that is running behind a PFSense server.

Now to my problem:
Sometimes the Kaniko Pods can't resolve the Hostname of the GitLab server. This leads to failed git pull and so to a failed build. I would rate the chance to fail by 60%, which is way too high for us. After retrying the build a few times, it will run without any problem.

The Kubernetes Cluster running the Gitlab CI is setup on CentOS 7. SELinux and FirewallD are disabled. All of the Hosts can resolve the GitLab Server. It is also not related to a specific Host Server, which is causing the problem. I have seen it fail on all of the 5 Servers including the Manager Server. Also I haven't seen this problem appear in other Pods. But the other Deployments in the cluster dont really do connections via DNS. I am sure that the Runner is able to access DNS at all, because it is pulling the Kaniko Image from gcr.io.

Has anyone ever seen this problem or knows a workaround?

  • I have already tried spawning Pods that only do DNS requests to the Domain. I didn't see a single fail.
  • Also I tried to Reboot the whole Cluster and Gitlab instance.
  • I tried to do a static overwrite of the DNS route in PFSense. Still same problem.

Here is my CI config:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo $REGISTRY_AUTH > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $REGISTRY_URL/$REGISTRY_IMAGE:$CI_JOB_ID
  only:
    - master

The following error happens:

Initialized empty Git repository in /builds/MYPROJECT/.git/
Fetching changes...
Created fresh repository.
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@git.mydomain.com/MYPROJECT.git/': Could not resolve host: git.mydomain.com
Jason Aller
  • 3,541
  • 28
  • 38
  • 38
xj32
  • 1
  • 1
  • 1
  • Maybe [this](https://stackoverflow.com/questions/50325932/gitlab-runner-docker-could-not-resolve-host) will help? – Crou Jul 18 '19 at 10:14

2 Answers2

5

We had same issue for couple of days. We tried change CoreDNS config, move runners to different k8s cluster and so on. Finally today i checked my personal runner and found that i'm using different version. Runners in cluster had gitlab/gitlab-runner:alpine-v12.3.0, when mine had gitlab/gitlab-runner:alpine-v12.0.1. We added line 

image: gitlab/gitlab-runner:alpine-v12.1.0

in values.yaml and this solved problem for us

IllidariCat
  • 111
  • 2
  • 8
2

There are a env for gitlab-runner that can solve this problem 

- name: RUNNER_PRE_CLONE_SCRIPT
  value: "exec command before git fetch ..."

for example: edit /etc/hosts

echo '127.0.0.1 git.demo.xxxx' >> /etc/hosts

or edit /etc/resolv.conf

echo 'nameserver 8.8.8.8' > /etc/resolv.conf

hope it works for you

dezhi.shen
  • 84
  • 4
  • My friend and I spent six days to find the way how to make gitlab kubernetes runner run. That was the only way to do it. – Ivan Bychkov Jan 22 '23 at 20:24