Docker on CentOS 7 - DNS not workin within containers

Question

I installed Docker on a CentOS 7 machine and DNS is not working within containers.

So, if I run nslookup google.com on my host, it resolves correctly. However, if I do docker container run busybox nslookup google.com I get:

Server:         8.8.8.8
Address:        8.8.8.8:53

Non-authoritative answer:
Name:   google.com
Address: 142.250.72.46

*** Can't find google.com: No answer

root@XXX [~]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc.)

Server:
 Containers: 57
  Running: 0
  Paused: 0
  Stopped: 57
 Images: 10
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.25.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 3.7GiB
 Name: cgh.cgh.net
 ID: X2YB:LILA:S5Q3:CLRJ:ACY7:ST4T:YRDD:3D3E:PLXV:DHGS:3IE2:QMIP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: akhtariev
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

root@xxx [~]# docker version
Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:43:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@XXX [~]# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "609282cb44a298bb8f69b424d4ba2b1e3ab391b37fc171d2ec032f592dc3afbb",
        "Created": "2021-05-23T14:52:38.789641794-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

I already tried setting the default DNS server to 8.8.8.8 and following various other troubleshooting guides but nothing has worked. It's worth mentioning that the server is provisioned by hostgator and has cPanel installed. I requested hostgator to disable anti virus and firewall. They do not provide support with Docker and could not help resolve my issue.

Any help is greatly appricated and please let me know if there is more information needed to help solve this issue.

UPDATE: This does not only happen on busybox but on other platforms as well. My main problem is that I cannot run a container of the image that is built from the following Dockerfile:

FROM python:3.9.5-buster

ENV HOME_PATH=/root
ENV PATH="${HOME_PATH}/.local/bin:${PATH}"
WORKDIR ${HOME_PATH}

COPY requirements.txt cron-job ${HOME_PATH}/
RUN apt-get update
RUN apt-get install -y cron
RUN useradd --create-home service
RUN pip install --user -r ${HOME_PATH}/requirements.txt

COPY . ${HOME_PATH}
RUN ls \
    && mv cron-job /etc/cron.d/cron-job \
    && chmod 0644 /etc/cron.d/cron-job \
    && crontab /etc/cron.d/cron-job \
    && touch ${HOME_PATH}/agl_history.log

CMD cron && tail -f ${HOME_PATH}/agl_history.log

docker-compose.yml:

version: "3.9"
services:
  agl-history:
    depends_on:
        - mariadb
    build: .
    restart: on-failure
    networks:
        - main
  mariadb:
    image: "mariadb:10.5"
    restart: on-failure
    environment:
        MYSQL_ROOT_PASSWORD: ${MARIADB_PASSW}
    ports:
        - 3306:3306
    volumes:
        - /var/lib/docker/volumes/add3-data:/var/lib/mysql
    networks:
        - main
networks:
    main:
        driver: bridge

This service works well and tested on my personal machine running the latest Ubuntu

UPDATE 2:

So, the recommendation in the comments worked to solve the problem with nslookup by doing docker run --rm busybox nslookup -type=a google.com

However, I still have the issue running the mentioned-above buster container. This is what happens:

Building agl-history
Sending build context to Docker daemon  918.5kB
Step 1/12 : FROM python:3.9.5-buster
 ---> a6a0779c5fb2
Step 2/12 : ENV HOME_PATH=/root
 ---> Using cache
 ---> 7aea834621d4
Step 3/12 : ENV PATH="${HOME_PATH}/.local/bin:${PATH}"
 ---> Using cache
 ---> dc16e6178688
Step 4/12 : WORKDIR ${HOME_PATH}
 ---> Using cache
 ---> 950712b33f21
Step 5/12 : COPY requirements.txt cron-job ${HOME_PATH}/
 ---> Using cache
 ---> 6923c1410731
Step 6/12 : RUN apt-get update
 ---> Using cache
 ---> aa4c4a4d25c4
Step 7/12 : RUN apt-get install -y cron
 ---> Running in 26c7517a05d0
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package cron
The command '/bin/sh -c apt-get install -y cron' returned a non-zero code: 100
ERROR: Service 'agl-history' failed to build : Build failed

Does this answer your question? [How should I interpret a "Can't find..." from nslookup inside a Docker busybox container?](https://stackoverflow.com/questions/52663711/how-should-i-interpret-a-cant-find-from-nslookup-inside-a-docker-busybox-c) — β.εηοιτ.βε, May 23 '21 at 20:26
Unfortunately, this doesn't only happen with busybox. In fact, busybox is only used for illustration. My main image is on buster. When I run a container of that image, it fails to get the dependencies from the internet because of this issue with DNS. I will update the question with the Dockerfile for this image — Roman Akhtariev, May 23 '21 at 20:31
Ok, but does the same `nslookup` works as soon s you drop both the `--dns` flag to `docker run` and `-type=a` or `-type=aaaa` to `nslookup` — β.εηοιτ.βε, May 23 '21 at 20:33
Thanks! You are right, it worked when I did `docker run --rm busybox nslookup -type=a google.com`. I will try to figure out how to use this knowledge to solve my original problem with installing dependencies in the buster container. If you have any suggestions, please let me know. I posted the contents of the Dockerfile as well as more information about that problem. Thanks again for the help! — Roman Akhtariev, May 23 '21 at 20:40
An error I can see in your Dockerfile is the fact that you `apt-get update` and `apt-get install` on two different `RUN` instructions, and as pointed in [the best practice](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run), this could cause you caching issues. Do `RUN apt-get update && apt-get install -y cron`, rather. — β.εηοιτ.βε, May 23 '21 at 20:50

score 0 · Accepted Answer · answered May 23 '21 at 21:21

You seems to be hitting a caching issue due to those two lines in your Dockerfile:

RUN apt-get update
RUN apt-get install -y cron

As pointed in the best practices for writing a Dockerfile, having apt-get update and apt-get install in two separate RUN instructions might make you face caching issues:

Using apt-get update alone in a RUN statement causes caching issues and subsequent apt-get install instructions fail. For example, say you have a Dockerfile:
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y curl
After building the image, all layers are in the Docker cache. Suppose you later modify apt-get install by adding extra package:
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y curl nginx
Docker sees the initial and modified instructions as identical and reuses the cache from previous steps. As a result the apt-get update is not executed because the build uses the cached version. Because the apt-get update is not run, your build can potentially get an outdated version of the curl and nginx packages.

Using RUN apt-get update && apt-get install -y ensures your Dockerfile installs the latest package versions with no further coding or manual intervention. This technique is known as “cache busting”.

^{Source: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run}

So your best bet would be to rather do:

RUN apt-get update && apt-get install -y \
        cron

^{Having the packages, one per line, on subsequent line, alphabetically sorted is another recommandation of the best practices.}

This solved my problem! Thank you so much! – Roman Akhtariev May 23 '21 at 22:47 — Roman Akhtariev, May 23 '21 at 22:47

Docker on CentOS 7 - DNS not workin within containers

1 Answers1