Kubernetes Pod is unable to mount volumes to GCP Filestore

Question

I am new to Kubernetes, and as a part of tutorial I have spun up a GKE cluster and a GCP Filestore instance.

Now I am trying to mount Grafana's volume to this Filestore instance. However, it is getting timed out. I am unable to decipher where the mistake lies. I need your help in addressing the issue.

PFB the output.

C:\Users\ak>kubectl describe pod/grafana-7c666cff94-vkgh4
Name:           grafana-7c666cff94-vkgh4
Namespace:      bc
Priority:       0
Node:           gke-bc-gke-cluster-bc-nodepool-9496e187-zsnw/10.51.0.5
Start Time:     Fri, 02 Sep 2022 16:21:28 +0530
Labels:         app=grafana
                pod-template-hash=7c666cff94
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/grafana-7c666cff94
Containers:
  grafana:
    Container ID:
    Image:          grafana/grafana:8.4.4
    Image ID:
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        250m
      memory:     750Mi
    Liveness:     tcp-socket :3000 delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:3000/robots.txt delay=10s timeout=2s period=30s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/lib/grafana from fileserver (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v7qjd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  fileserver:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  fileserver-claim
    ReadOnly:   false
  kube-api-access-v7qjd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    43m                   default-scheduler  Successfully assigned bluecopa/grafana-7c666cff94-vkgh4 to gke-bc-gke-cluster-bc-nodepool-9496e187-zsnw
  Warning  FailedMount  4m15s (x11 over 40m)  kubelet            MountVolume.SetUp failed for volume "fileserver" : mount failed: exit status 1
Mounting command: /home/kubernetes/containerized_mounter/mounter
Mounting arguments: mount -t nfs 10.168.189.130:/bc_fs /var/lib/kubelet/pods/cf44b980-7461-4c0e-a32f-673588160692/volumes/kubernetes.io~nfs/fileserver
Output: Mount failed: mount failed: exit status 32
Mounting command: chroot
Mounting arguments: [/home/kubernetes/containerized_mounter/rootfs mount -t nfs xx.xx.xx.xx:/bc_fs /var/lib/kubelet/pods/cf44b980-7461-4c0e-a32f-673588160692/volumes/kubernetes.io~nfs/fileserver]
Output: mount.nfs: Connection timed out
  Warning  FailedMount  3m16s (x12 over 37m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[fileserver], unattached volumes=[fileserver kube-api-access-v7qjd]: timed out waiting for the condition
  Warning  FailedMount  59s (x7 over 41m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[fileserver], unattached volumes=[kube-api-access-v7qjd fileserver]: timed out waiting for the condition

PV.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: fileserver
  namespace: bluecopa
spec:
  capacity:
    storage: 200Gi
  accessModes:
  - ReadWriteMany
  nfs:
    path: /bc_fs
    server: xx.xx.xx.xx

PVC.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fileserver-claim
  namespace: bluecopa
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: ""
  volumeName: fileserver
  resources:
    requests:
      storage: 100Gi

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: bluecopa
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:8.4.4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: fileserver
      volumes:
        - name: fileserver
          persistentVolumeClaim:
            claimName: fileserver-claim

Answer 1

While using the volume mounts in pods we need to watch out for security context Use the securitycontext as follows in deployment file

securityContext:
  runAsUser: 0

Use the following security context in the deployment file

This will help you out to mount the volume without any issues.

For more information check out this documents Doc1 & Doc2

Here is the output of deployment pod