6

I have observed that the pods in my cluster is not able to install the packages when exec to the pod . While debugging i have realized that its due to /etc/resolv.conf entries.

The /etc/resolv.conf entry from one of the pod is :

nameserver 192.168.27.116
search ui-container.svc.cluster.local svc.cluster.local cluster.local 192.168.27.116.nip.io
options ndots:5

Here if i remove a entry 192.168.27.116.nip.io from the resolv.conf of all the master, worker nodes then the pods will be able to connect to internet and apt-get update and apt-get install works. This is only a temporary workaround because it is not recommended to update the resolv.conf. because i have observed that resolv.conf contents gets re-set to original upon reboot of the nodes.

Is it due to options ndots:5 in the /etc/resolv.conf ?

How can i fix this?

Jonas
  • 121,568
  • 97
  • 310
  • 388
Rakesh Kotian
  • 175
  • 3
  • 20

3 Answers3

4

As a quick-fix, you could leverage dnsConfig from pod Spec to override default dns configuration, more details: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config

Ottovsky
  • 2,068
  • 15
  • 22
3

No, it's because of the nip.io address in your resolv.conf. It's a special domain that's used for reflection and mocking. It's not the ndots:5.

Gustavo Kawamoto
  • 2,665
  • 18
  • 27
  • Thanks for the info. The entries in the `/etc/resolv.conf` are auto-generated. How can i solve the issue then? – Rakesh Kotian Sep 03 '20 at 09:03
  • how you installed cluster? – Vit Sep 03 '20 at 09:11
  • 1
    Seems that the domain set for the cluster is the `192.168.27.116.nip.io`. One option would be change the default domain for your cluster so it wouldn't use the `nip.io`. That would fix every pod you have in your cluster. You can also change the `search` [per pod](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config), using `dnsConfig`. – Gustavo Kawamoto Sep 03 '20 at 09:12
  • Yes cluster hostname is set to `192.168.27.116.nip.io` since it eliminated our need to edit the /etc/hosts entry for the routes. Thanks for your suggestions for fix. Im looking for alternate solution other any removing `nip.io` or setting search for pod. – Rakesh Kotian Sep 03 '20 at 10:14
  • @Vitalii the cluster was installed using https://github.com/openshift/openshift-ansible/tree/release-3.11 – Rakesh Kotian Sep 03 '20 at 10:15
  • 1
    The issue is that `nip.io` is a wide ranging domain. Because of that, anything that you throw at them will be forwarded to `192.168.27.116`. If you can override the cluster settings I guess you could mess around with the order domains are resolved (such as including top domains before your `192.168.27.116.nip.io`), but that's a long shot and could potentially leak information from your cluster and/or slow things down. Good luck! – Gustavo Kawamoto Sep 03 '20 at 10:24
  • What I don't understand is why the "search" needs to be used, if the URL for apt server is correct? What I understand the search is options that will be appended to the hostname just in the case that the hostname cannot be resolved. – Chayne P. S. Sep 11 '20 at 21:21
  • You're getting it right @ChayneP.S.. The problem here is that the `resolv.conf` is appending for long domains as well, because of the `ndots:5` option. That's a configuration set for the search to be relayed straight to public DNS when there's more than 5 dots in the domain, and setting it lower or higher could be problematic because it could mess with the cluster internal name resolution. – Gustavo Kawamoto Sep 14 '20 at 14:46
  • @GustavoKawamoto Thank you for explaining. It's clear to me now. – Chayne P. S. Sep 14 '20 at 15:00
2

Adding to above answers, usually resolve.conf follows following pattern

nameserver <local domain server provided by docker>
search <pod namespace>.svc.cluster.local svc.cluster.local cluster.local <host/node's actual DNS>
options ndots:5

Last part of the DNS <host/node's actual DNS> is usually your LAB's or system wide DNS that is part of your network. So you may have override with DNS policy or update the same in your lab. nip.io is a FQDN maker where there is a need, so you may come up with a local DNS name that resolves within your LAB instead of relying on that.

user2039152
  • 146
  • 8