Can't resolve dns from inside k8s pod

Question

In dnsutils pod exec ping stackoverflow.com

/ # ping stackoverflow.com
ping: bad address 'stackoverflow.com'

The /etc/resolve.conf file looks fine from inside the pod

/ # cat /etc/resolv.conf 
nameserver 10.96.0.10
search weika.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

10.96.0.10 is the kube-dns service ip:

[root@test3 k8s]# kubectl -n kube-system get service
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                     AGE
kube-dns                    ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP      75d

core dns

[root@test3 k8s]# kubectl -n kube-system get  pod -o wide | grep core
coredns-6557d7f7d6-5nkv7                             1/1     Running   0          10d   10.244.0.14    test3.weikayuninternal.com   <none>           <none>
coredns-6557d7f7d6-gtrgc                             1/1     Running   0          10d   10.244.0.13    test3.weikayuninternal.com   <none>           <none>

when I change the nameserver ip to coredns ip. resolve dns is ok.

/ # cat /etc/resolv.conf 
nameserver 10.244.0.14
#nameserver 10.96.0.10
search weika.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # ping stackoverflow.com
PING stackoverflow.com (151.101.65.69): 56 data bytes
64 bytes from 151.101.65.69: seq=0 ttl=49 time=100.497 ms
64 bytes from 151.101.65.69: seq=1 ttl=49 time=101.014 ms
64 bytes from 151.101.65.69: seq=2 ttl=49 time=100.462 ms
64 bytes from 151.101.65.69: seq=3 ttl=49 time=101.465 ms
64 bytes from 151.101.65.69: seq=4 ttl=49 time=100.318 ms
^C
--- stackoverflow.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 100.318/100.751/101.465 ms
/ # 

Why is it happening?

Answer 1

You have not mentioned how kubernetes was installed. You should restart coredns pods using below command.

kubectl -n kube-system rollout restart deployment coredns

Answer 2

This might only apply to you if there was trouble during either your initial installation of microk8s or enablement of the dns addon, but it might still be worth a shot. I've invested so much gd time in this I couldn't live with myself if I didn't at least share to help that one person out there.

In my case, the server I provisioned to set up a single-node cluster was too small - only 1GB of memory. When I was setting up microk8s for the first time and enabling all the addons I wanted (dns, ingress, hostpath-storage), I started running into problems that were remedied by just giving the server more memory. Unfortunately though, screwing that up initially seems to have left the addons in some kind of undefined, partially initialized/configured state, such that everything appeared to be running normally as best I could tell (i.e. CoreDNS was deployed and ready, and the kube-dns service showed CoreDNS's ClusterIP as it's backend endpoint) but none of my pods could resolve any DNS names, internal or external to the cluster, and I would get these annoying event logs when I ran kubectl describe <pod> suggesting there was a DNS issue of some kind.

What ended up fixing it is resetting the cluster (microk8s reset --destroy-storage) and then re-enabling all my addons (microk8s enable dns ingress hostpath-storage) now that I had enough memory to do so cleanly do so. After that, CoreDNS and the kube-dns service appeared ready just like before, but DNS queries actually worked like they should from within the pods running in the cluster.

tl;dr; - Your dns addon might have have been f'd up during cluster installation. Try resetting your cluster, re-enabling the addons, and re-deploying your resources.