8

I have a GitLab installation running in Kubernetes, and suddenly, my connections to dind have stopped working. This problem started appearing in a single project out of ~30 and is working in the other ones, and no change has been made.

The builds give the following errors:

*** WARNING: Service runner-c542f8fe-project-3-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-c542f8fe-project-3-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2018-08-13T08:40:53.274661600Z mount: permission denied (are you root?)
2018-08-13T08:40:53.274713900Z Could not mount /sys/kernel/security.
2018-08-13T08:40:53.274730800Z AppArmor detection and --privileged mode might break.
2018-08-13T08:40:53.275949300Z mount: permission denied (are you root?)

*********

I am running the container privileged, as can be seen in my /etc/gitlab-runner/config.toml:

metrics_server = ":9252"
concurrent = 10
check_interval = 30

[[runners]]
  name = "mothy-jackal-gitlab-runner-bb76cb464-7fq6z"
  url = "[redacted]"
  token = "[redacted]"
  executor = "kubernetes"
  [runners.cache]
  [runners.kubernetes]
    host = ""
    image = "ubuntu:16.04"
    namespace = "gitlab"
    namespace_overwrite_allowed = ""
    privileged = true
    cpu_request = "100m"
    memory_request = "128Mi"
    service_cpu_request = "100m"
    service_memory_request = "128Mi"
    service_account_overwrite_allowed = ""
    [runners.kubernetes.volumes]

The only other solution I've found that don't pertain to making sure that the runner is privileged is this one. I've tried setting the variables in my .gitlab-ci.yaml to this:

variables:
  DOCKER_HOST: "tcp://docker:2375"
  DOCKER_DRIVER: overlay

The error remains the same.

Worth noting is the output of these following commands, in accordance with the other post:

bash-4.3# find /lib/modules/`uname -r`/kernel/ -type f -name "overlay*"
find: /lib/modules/4.4.111-k8s/kernel/: No such file or directory
bash-4.3# lsmod | grep overlay
overlay                45056 12

Note the "No such file or directory" error.

I'm stumped, and with my builds failing in the registry stage, I can't make releases. Any pointers as of where to go?

Thanks.

EDIT
It's not a solution, but I noticed that this occurred because I had set a dedicated runner to this project. Once I removed that, it worked again. Not a fix, but important info to anyone having the same issue.

Helge Talvik Söderström
  • 2,573
  • 4
  • 35
  • 48
  • Hey, did you ever figure out what was it? We are seeing this happening more and more recently. Our working hypothesis is that it might be related to the version of Docker for which dind service was compiled and the Docker running gitlab-runner. Another idea was a race in how Docker handles udev or device mapper interaction, perhaps there's some race there, related to mounting the socket file. – wvxvw Jan 24 '19 at 16:18
  • Hey wvxvw! I stopped investigating this since I realized it was connected to having a dedicated runner for that project. When I removed that and allowed shared runners, the problem went away, as per my edit. Sorry I can't help you further! – Helge Talvik Söderström Jan 24 '19 at 16:25
  • Heh, thanks, in our case we can only have dedicated runner... :) – wvxvw Jan 24 '19 at 16:35

0 Answers0