Some pods not resolving via CoreDNS in k3s

Question

I am running k3s cluster on RPis with Ubuntu Server 20.04 and have a drive mounted onto the master node to serve as shared NFS storage.

For some reason I can only access the server using the ClusterIP of the service, but no the name. CoreDNS is running and the logs suggest that it's working fine, but it's not used by the test pod that I try to mount the NFS share onto.

There is a question for Raspbian case, but I am running Ubuntu, not sure if using legacy iptables as suggested there will work or break things further.

Any help is appreciated! Details below.

NFS server manifests:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-server-pod
  namespace: storage
  labels:
        nfs-role: server
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
        nfs-role: server
  template:
    metadata:
      labels:
        nfs-role: server
    spec:
      nodeSelector:
        storage: enabled
      containers:
        - name: nfs-server-container
          image: ghcr.io/two-tauers/nfs-server:0.1.14
          securityContext:
            privileged: true
          args:
            - /storage
          volumeMounts:
          - name: storage-mount
            mountPath: /storage
      volumes:
      - name: storage-mount
        hostPath:
          path: /storage

---
kind: Service
apiVersion: v1
metadata:
  name: nfs-server
  namespace: storage
spec:
  selector: 
        nfs-role: server
  type: ClusterIP
  ports:
    - name: tcp-2049
      port: 2049
      protocol: TCP
    - name: udp-111
      port: 111

❯ kubectl logs pod/nfs-server-pod-ccd9c4877-lgd4x -n storage
 * Exporting directories for NFS kernel daemon...
   ...done.
 * Starting NFS kernel daemon
   ...done.
Setting up watches.
Watches established.

❯ kubectl get service/nfs-server -n storage
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
nfs-server   ClusterIP   10.43.223.27   <none>        2049/TCP,111/TCP   102m

Test pod, not in a running state:

---
kind: Pod
apiVersion: v1
metadata:
  name: test-nfs
  namespace: storage
spec:
  volumes:
    - name: nfs-volume
      nfs: 
        server: nfs-server # doesn't work
        ### server: nfs-server.storage.svc.cluster.local # doesn't work
        ### server: 10.43.223.27 # works
        path: /
  containers:
    - name: test
      image: alpine
      volumeMounts:
        - name: nfs-volume
          mountPath: /var/nfs
      command: ["/bin/sh"]
      args: ["-c", "while true; do date >> /var/nfs/dates.txt; sleep 5; done"]

Events:

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    51m                   default-scheduler  Successfully assigned storage/test-nfs to sauron
  Warning  FailedMount  9m26s (x14 over 48m)  kubelet            MountVolume.SetUp failed for volume "nfs-volume" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs nfs-server:/ /var/lib/kubelet/pods/d1523d5f-60ec-4990-ae61-369a26b7a6f4/volumes/kubernetes.io~nfs/nfs-volume
Output: mount.nfs: Connection timed out
  Warning  FailedMount  8m12s (x15 over 49m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[nfs-volume], unattached volumes=[nfs-volume kube-api-access-pc9bk]: timed out waiting for the condition
  Warning  FailedMount  3m44s (x5 over 46m)   kubelet  Unable to attach or mount volumes: unmounted volumes=[nfs-volume], unattached volumes=[kube-api-access-pc9bk nfs-volume]: timed out waiting for the condition

At the same time, nslookup works and coredns does produce a lookup log:

---
apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: storage
spec:
  containers:
  - name: dnsutils
    image: k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

❯ kubectl exec -i -t dnsutils -n storage -- nslookup nfs-server && kubectl logs pod/coredns-84c56f7bfb-66s84 -n kube-system | tail -1
Server:     10.43.0.10
Address:    10.43.0.10#53

Name:   nfs-server.storage.svc.cluster.local
Address: 10.43.223.27

[INFO] 10.42.1.77:37937 - 21822 "A IN nfs-server.storage.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.000980753s

❯ kubectl exec -i -t dnsutils -n storage -- nslookup kubernetes.default.svc.cluster.local && kubectl logs pod/coredns-84c56f7bfb-66s84 -n kube-system | tail -1
Server:     10.43.0.10
Address:    10.43.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.43.0.1

[INFO] 10.42.1.77:59077 - 58999 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.000496515s

Some pods not resolving via CoreDNS in k3s

0 Answers0