1

I wrote a simple script in my docker ENTRYPOINT to use dnsmasq

if [ ! -f /etc/resolv.dnsmasq ];then
   cp /etc/resolv.conf /etc/resolv.dnsmasq
   sed -i 's/^nameserver.*/nameserver 127.0.0.1/' /etc/resolv.conf
   dnsmasq -r /etc/resolv.dnsmasq
else
   dnsmasq -r /etc/resolv.dnsmasq
fi

My logic is simple, the first time k8s starts my docker contain, it will update nameserver in /etc/resolv.conf to the correct value and my script will copy it to /etc/resolv.dnsmasq and change nameserver in /etc/resolv.conf to 127.0.0.1 to use dnsmasq service.

When docker restarts because /etc/resolv.dnsmasq remains so dnsmasq will just start. It works most of time but when k8s restarts container due to health check failure it will fail to work. When that happen my resolv.dnsmasq has nameserver 127.0.0.1,

/var/www/html # cat /etc/resolv.dnsmasq
nameserver 127.0.0.1
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:2 timeout:2

From the comment and answer I got so far I realize it was because when the container was recreated then I have a modified /etc/resolv.conf and no /etc/resolv.dnsmasq. so the then part is run.

But why will I have nameserver 127.0.0.1 in /etc/resolv.conf if it is recreated from the image ?

How can I deal with that and make dnsmasq work ?

--- update---

Through our investigation it seems that sometimes Kubernetes will kill the running docker container and start a new container from image, but with the pause container exists the modified /etc/resolve.conf also exists. When that happens the restarted container does not have /etc/resolv.dnsmasq but it shares the /etc/resolv.conf that was already modified nameserver 127.0.0.1

I am not sure the whole logic behind the pause container. What is the use of a pause image in Kubernetes? said "The pause container holds the network namespace for the pod." So does that mean /etc/resolv.conf remains in my case? Can someone help to answer it ?

Qiulang
  • 10,295
  • 11
  • 80
  • 129
  • It's not that the file is deleted. Files that are created inside the container continue to exist unless the container needs to be recreated from the image. If it needs to be recreated, only the files that exist on the image will exist. – yivi Dec 24 '21 at 09:25
  • Hi I saw you removed my second question. So I plan to rewrite my question to ask it because that is the main point of my question. And according to you it is because container is recreated. Is that okay with you ? – Qiulang Dec 24 '21 at 09:31
  • Hi I have updated my question. But why will I have a modified `/etc/resolv.conf` if it is recreated from the image ? – Qiulang Dec 24 '21 at 09:48
  • Have you considered using `postStart` hook in your Deployment? You would move your `ENTRYPOINT` script there. It should fix your problem in k8s cluster but it requires to change Dockerfile. Let me know what you think and if you are planning to use this container standalone or only in cluster. – mdobrucki Dec 24 '21 at 10:39
  • In Kubernetes, do you actually need to run dnsmasq? Kubernetes has [its own DNS system](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) and [a way to define additional names in it](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). – David Maze Dec 24 '21 at 11:34
  • That is another question. I had thought I don't need dnsmasq either but it turns out that in some Kubernetes environment, the DNS resolve keeps failed and when I introduced dnsmasq the problem was solved! Check my question here https://stackoverflow.com/questions/67529042/what-is-the-difference-between-alpine-docker-image-and-busybox-docker-image – Qiulang Dec 24 '21 at 12:10
  • The biggest downside of Alpine-based images is that Alpine has a different implementation of the core C library and sometimes there are compatibility issues. This sounds like an outright bug more than a compatibility issue, but you might try rebuilding your image against something based on the more standard GNU libc (typically a Debian or Ubuntu base) and see if that helps. – David Maze Dec 24 '21 at 12:17
  • @Qiulang邱朗 have you considered my suggestion? – mdobrucki Dec 24 '21 at 12:33
  • @mdobrucki Not yet. I need to fix this particular question first. – Qiulang Dec 24 '21 at 12:37
  • Can you provide your Dockerfile? – mdobrucki Dec 24 '21 at 12:56
  • I still hit DNS failure intermittently without dnsmasq setup correctly and I find [this article](https://betterprogramming.pub/why-i-will-never-use-alpine-linux-ever-again-a324fd0cbfd6) explains the reason "musl (by design) doesn't support DNS-over-TCP... when a a single UDP packet (512 bytes) is not enough to resolve hostnames" – Qiulang Mar 14 '23 at 14:15

1 Answers1

0

...when k8s restarts container due to health check failure it always fail to work

Different from docker restart, K8s restart container with new instance, anything you saved in the killed container will not be there. For your file(s) to persist thru K8s restart, you need to save them in persistent volume (eg. NFS, storage supplied by your cloud provider etc).

gohm'c
  • 13,492
  • 1
  • 9
  • 16
  • Hi I updated my question, can you take a look again ? – Qiulang Dec 24 '21 at 09:44
  • Checkout the updated answer. – gohm'c Dec 24 '21 at 09:46
  • But why will I have a modified /etc/resolv.conf if it is recreated from the image ? – Qiulang Dec 24 '21 at 09:49
  • That is exactly my question! I don't think /etc/resolv.conf has 127.0.0.1 in image. But is it ? I am not 100% sure now. – Qiulang Dec 24 '21 at 09:52
  • Not sure what you mean, note by default a pod inherits the name resolution configuration from the node. – gohm'c Dec 24 '21 at 10:03
  • Now my /etc/resolv.dnsmasq has `nameserver 127.0.0.1`. I don't know how that could happen. – Qiulang Dec 24 '21 at 10:08
  • Can you post the **first** version of `/etc/resolv.conf` **before** you copy into another file? Note all comments made here is for **k8s** and **not** docker. – gohm'c Dec 24 '21 at 10:16
  • I can `if ! grep -q "127.0.0.1" /etc/resolv.conf && [ ! -f /etc/resolv.dnsmasq ]; then ... ` but that won't solve my problem, i.e. to use dnsmasq – Qiulang Dec 24 '21 at 10:22
  • Can you do a `cat /etc/resolv.conf ` before your bash script, then use `kubectl logs ` to see the original content of resolv.conf. – gohm'c Dec 24 '21 at 10:32
  • By the way, the `cat` is to be run during pod start up like `initContainers`, **not** in your Dockerfile ENTRYPOINT/CMD or docker-entrypoint.sh. – gohm'c Dec 24 '21 at 10:34
  • Can you check my update ? – Qiulang Dec 28 '21 at 07:33