Mounting a NFS volume by a OpenShift 3.11 PersistentVolume: mount.nfs: mounting failed, reason given by server: No such file or directory

Question

In our OpenShift 3.11 cluster, we are trying to use NFS through a PersistentVolume and a NFS volume previously created on a external NFS storage (a Isilon Storage). We created and applied succesfully the PersistentVolume and the PersistentVolumeClaim on the Kubernetes/OpenShift Layer. The PVC binds the PV correctly, but when checking the Deployment events we face an error in the mounting NFS phase.

PersistentVolume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: tool1pv
spec:
  capacity:
    storage: 100Gi 
  accessModes:
  - ReadWriteOnce 
  nfs: 
    path: /tool1shareenv1
    server: tommytheserver.companydomain.priv
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: tool1claimenv1
    namespace: ocpnamespace1

PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tool1claimenv1
spec:
 accessModes:
   - ReadWriteOnce
 resources:
  requests:
    storage: 100Gi
 volumeName: tool1pvenv1

When checking the Development Events, we see a "No such file":

MountVolume.SetUp failed for volume "tool1pvenv1" : mount failed:
 exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/f1cb1291-fe12-01ea-bb92-0050123aa39be/volumes/kubernetes.io~nfs/tool1pvenv1 --scope -- mount -t nfs tommytheserver.companydomain.priv:/tool1shareenv1
/var/lib/origin/openshift.local.volumes/pods/f1cb9191-fe73-11ea-bb92-005056ba12be/volumes/kubernetes.io~nfs/tool1pvenv1d Output: Running scope as unit run-74039.scope. **mount.nfs: mounting tommytheserver.companydomain.priv:/tool1env1 failed, reason given by server: No such file or directory**

We investigated the server and the path fields and tried different varations such as:

PersistentVolumeVersion2:

 apiVersion: v1
kind: PersistentVolume
metadata:
  name: tool1pv
spec:
  capacity:
    storage: 100Gi 
  accessModes:
  - ReadWriteOnce 
  nfs: 
    path: /tool1shareenv1
    server: tommytheserver.companydomain.priv/tool1shareenv1
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: tool1claimenv1
    namespace: ocpnamespace1

but we're still faceing the same No such file error.

How can we troubleshoot it ?

Does that `/tool1shareenv1` share exist? Is it properly allowing connections from your nodes? Can you try and mount that share manually, on one of your nodes? — SYN, Sep 25 '20 at 06:19

score 3 · Accepted Answer · answered Sep 24 '20 at 16:08

Normally to troubleshoot something like this I would...

Double check that your share path actually exists
Get the IP address of your node where your pod ran and ssh onto it. You can get the IP like this:

kubectl get pod <podname> -o wide -n namespace

Then I would make sure I can connect to the nfs server where the share exists:

telnet <nfs server> port

Following that I would run dmesg to see mounting related errors
I would try to mount the volume myself using the same arguments your error is showing. ie-

mount -t nfs tommytheserver.companydomain.priv:/tool1shareenv1

It is difficult to provide a specific answer without seeing the results of these troubleshooting steps. But, that is the approach I would take.

The final mount command showed the mistake: we were entering the wrong value in the path field of the PV. In more details, we entered the path we use to mount the volume while it requests the "Share Name" (in our case was something like "/vol/NFS1234X56"). We corrected the field and the error disappeared. — lsambo, Sep 25 '20 at 10:40

Mounting a NFS volume by a OpenShift 3.11 PersistentVolume: mount.nfs: mounting failed, reason given by server: No such file or directory

1 Answers1