Gitlab runner docker Could not resolve host

Question

Im using 2 containers on my Ubuntu OS: Gitlab-ce and gitlab-runner

Containers names are: gitlab_gitlab_1 and gitlab_gitlab-runner_1

I access to my gitlab app via gitlab.localhost.com:801

I register successfully a runner with this command:

docker exec -it gitlab_gitlab-runner_1 gitlab-runner register --non-interactive --url http://gitlab_gitlab_1 --registration-token _wgMgEx3nBocYQtoi83c --executor docker --docker-image alpine:latest

Then, when I start the job, I got this error message:

Running with gitlab-runner 10.7.1 (b9bba623)
  on 589a617ee407 12ba77f7
Using Docker executor with image alpine:latest ...
Pulling docker image alpine:latest ...
Using docker image sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353 for alpine:latest ...
Running on runner-12ba77f7-project-1-concurrent-0 via 01196621a827...
Cloning repository...
Cloning into '/builds/root/test'...
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.localhost.com/root/test.git/': Could not resolve host: gitlab.localhost.com
ERROR: Job failed: exit code 1

In both containers, I can access to the hostname gitlab.localhost.com. I think the issue comes from the image alpine which can not resolve the host.

How can I fix that?

Thanks

Edit 1

docker-compose.yml

version: '3'
services:
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.localhost.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.localhost.com'
    ports:
      - '801:80'
      - '443:443'
      - '22:22'
    volumes:
      - '/srv/gitlab/config:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'
    networks:
      - 'default'

  gitlab-runner:
    image: 'gitlab/gitlab-runner:latest'
    depends_on:
      - 'gitlab'
    restart: always
    volumes:
      - '/srv/gitlab-runner/config:/etc/gitlab-runner'
      - '/var/run/docker.sock:/var/run/docker.sock'
    networks:
      - 'default'
    links:
      - 'gitlab:gitlab.localhost.com'

networks:
  default:
    driver: 'bridge'

Edit 2

docker-compose.yml

version: '3'
services:
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.localhost.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.localhost.com'
    ports:
      - '801:80'
      - '443:443'
      - '22:22'
    volumes:
      - '/srv/gitlab/config:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'
    networks:
      default:
        aliases:
          - 'gitlab.localhost.com'

  gitlab-runner:
    image: 'gitlab/gitlab-runner:latest'
    depends_on:
      - 'gitlab'
    restart: always
    volumes:
      - '/srv/gitlab-runner/config:/etc/gitlab-runner'
      - '/var/run/docker.sock:/var/run/docker.sock'
    networks:
      - 'default'

networks:
  default:
    driver: 'bridge'

have you tried to create a docker network for these containers so they can be linked together ? — Mostafa Hussein, May 14 '18 at 08:49
Yes I also created a user defined network with the default bridge driver. So they are in the same network. — fliim, May 14 '18 at 09:19
In this case you access it within docker using the service name which i assume it will be "gitlab" where you can access it like this: http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab/root/test.git/, or you can define a network alias to it or using this too gitlab_gitlab_1 — Mostafa Hussein, May 14 '18 at 09:31
How can i specify this ? I already register the runner using the container name gitlab_gitlab_1, didnt work. — fliim, May 14 '18 at 09:37
Related to https://gitlab.com/gitlab-org/gitlab-runner/issues/1846 ? — Tarun Lalwani, May 16 '18 at 17:42
Thank, I post the fix to my issue thanks to the post you linked. — fliim, May 22 '18 at 07:19

score 40 · Accepted Answer · edited Aug 20 '21 at 22:47

40

Thanks to Tarun Lalwan link and according to Joyce Babu post, there are an undocumented option from the gitlab runner repos in the [runners.docker] section

network_mode : Add container to a custom network

So I have to set this option with my network name in the config.toml like

[[runners]]
  ...
  [runners.docker]
    ...
    network_mode = "gitlab_default"

OR when create the runner from command line

docker exec -it gitlab_gitlab-runner_1 gitlab-runner register \
--non-interactive \
--url http://gitlab_gitlab_1 \
--registration-token _wgMgEx3nBocYQtoi83c \
--executor docker \
--docker-image alpine:latest \
--docker-network-mode gitlab_default

edited Aug 20 '21 at 22:47

miku

181,842
47
306
310

answered May 22 '18 at 07:16

fliim

2,059
4
18
31

Thank you so much for this! I was trying to figure this out for 2 hours... – jantobola Nov 15 '18 at 17:19
9

Thank you! I also wanted to add that "gitlab_default" is specific to @fliim's docker-compose.yml. Remember the network name is the name of the service plus the name given inside the docker-compose.yml. For example, mine is "infrastructure" and the network is "backend", so that network_mode line is `network_mode = "infrastructure_backend"`. – bmauter Dec 30 '18 at 23:19
3

Thank you. You can list your networks by using `docker network ls` – Apr 08 '20 at 03:30
GOD BLESS YOU!! It's work! – luchanos Nov 21 '22 at 11:40

score 27 · Answer 2 · answered Oct 16 '20 at 22:30

I know this thread is old, but I saw everywhere threads pointing to this one, but did'nt solve for me.

I got the same error message :

fatal: unable to access 'http://gitlab.maison.fr:82/angular/test1.git/': Could not resolve host: gitlab.maison.fr

Adding network_mode = "host" to config.toml solve the problem.

My config.toml below on my local private network :

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "mars"
  url = "http://gitlab.maison.fr:82/"
  token = "TCfHAheTUdWMU2-fFNxK"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "gitlab/gitlab-runner:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mode = "host"

Freman Zhang · Answer 3 · 2021-10-16T14:50:32.180

6

This is working for me without define network in docker compose. Just add a extra_hosts record in gitlab-runner config just likes we add A record in other OS.

sudo vim /etc/gitlab-runner/config.toml

  [runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
extra_hosts = ["xxx.xxx.com.cn:192.168.10.110"]

https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section

edited Oct 16 '21 at 14:50

answered Oct 16 '21 at 10:19

Freman Zhang

491
6
6

1

Thank you! I've spent days trying to fix this and this was the only one that worked. Must be a newer problem because all of the answers seem to be from ~2017 – logan May 10 '22 at 17:23

score 2 · Answer 4 · answered Oct 30 '18 at 08:16

In case this helps others looking for this..

Same problem but GitLab and GitLab Runner run on different machines in LAN. DNS is working and ping gitlab works, except inside dockers:

Reproduce problem:

$ sudo docker run -it alpine ping gitlab
ping: bad address 'gitlab'
^C

But works with DNS given:

$ sudo docker run -it --dns=172.168.0.1 alpine ping gitlab
PING gitlab (172.168.0.5): 56 data bytes
64 bytes from 172.168.0.5: seq=0 ttl=63 time=0.536 ms
^C

Config actual LAN DNS for docker.

Edit /etc/docker/daemon.json on the GitLab Runner (file did not exist yet) with contents:

{
    "dns": ["172.168.0.1", "1.1.1.1"]
}

Test again, now OK:

$ sudo docker run -it --dns=172.168.0.1 alpine ping gitlab
PING gitlab (172.168.0.5): 56 data bytes
64 bytes from 172.168.0.5: seq=0 ttl=63 time=0.455 ms
64 bytes from 172.168.0.5: seq=1 ttl=63 time=0.905 ms
^C

If this is not how its supposed to be done, i'd be happy to hear.
If this problem shouldnt aught to exist in the first place, i'd be happy to hear as well. I was surprised to not find much references online to this problem for GitLab Runner..

score 0 · Answer 5 · answered May 14 '18 at 09:50

0

As I can see you have defined a network already which means that both gitlab and gitlab-runner are in the same network. you can verify that by using docker inspect. so you need to remove links as you don't need it.

In order to set a network alias you need to change the network part at gitlab service to the following:

gitlab:
  ...
  networks:
    default:
      aliases:
        - gitlab.localhost.com

References:

Compose file version 3 reference

answered May 14 '18 at 09:50

Mostafa Hussein

11,063
3
36
61

I update my docker-compose.yml file as you suggest, run `docker-compose down` and `docker-compose up -d`. Nothing changed. For remind, gitlab-runner container can access gitlab.localhost.com but not the job. I think its inside the alpine:latest container run by the runner that cant access to this hostname – fliim May 14 '18 at 09:58
you can verify that gitlab container has the alias. and also you can log into gitlab-runner container and try to run ping or wget or curl on each gitlab's alias you will find through docker inspect. – Mostafa Hussein May 14 '18 at 10:05
The alias work inside the gitlab-runner container, as I said. But not in the job run from gitlab. – fliim May 14 '18 at 10:10
the error you posted seems like a normal git clone $URL, can you do the same manually within the runner container ? – Mostafa Hussein May 14 '18 at 12:40
`git clone http://gitlab.localhost.com/root/test.git` works from the runner container – fliim May 14 '18 at 13:38

score 0 · Answer 6 · answered Oct 05 '21 at 13:09

0

I had this problem these days, after trying all the proposed solutions of the various threads, I solved it in the following way:

pkill docker
iptables -t nat -F
ifconfig docker0 down
service docker restart

thanks to My docker container has no internet

answered Oct 05 '21 at 13:09

Riccardo Venturini

400
2
6

score 0 · Answer 7 · answered Oct 18 '21 at 22:46

Network-wise a container is like a separate server with it's own ip address, network routes and DNS resolution. The only exception is to run the container on the host network.

That said, you probably need to figure out how DNS resolution works for the host machine (maybe you added something to /etc/hosts?) and then do the same inside the container.

Or use the exception and run on the host network.

score 0 · Answer 8 · answered Dec 07 '22 at 13:20

For anyone trying to deploy a runner using a docker executor in a private network with its own dns setup (i.e. having their own dns server), I tried the above solution in specifying the docker network in config.toml and it didn't work. Even though I could resolve gitlab.com from the host and the runner container I kept getting the above mentioned error. So the job container couldn't resolve gitlab.com. In the end what worked for me:

[[runners]]
  ...
  [runners.docker]
    ...
    network_mode = "host"

score 0 · Answer 9 · answered Jun 15 '23 at 05:25

0

i gat same problem, in my issue server proxy was open and just disable it. #systemctl status firewalld #systemctl stop firewalld

answered Jun 15 '23 at 05:25

Taner Mutlu

1

Gitlab runner docker Could not resolve host

9 Answers9

Reproduce problem:

Config actual LAN DNS for docker.

Linked