14

I have a Kubernetes cluster with some pods deployed (DB, Frontend, Redis). A part that I can't fully grasp is what happens to the PVC after the pod is deleted.

For example, if I delete POD_A which is bound to CLAIM_A I know that CLAIM_A is not deleted automatically. If I then try to recreate the POD, it is attached back to the same PVC but the all the data is missing.

Can anyone explain what happens, I've looked at the official documentation but not making any sense at the moment.

Any help is appreciated.

Rutnet
  • 1,533
  • 5
  • 26
  • 48

1 Answers1

19

PVCs have a lifetime independent of pods. If PV still exists it may be because it has ReclaimPolicy set to Retain in which case it won't be deleted even if PVC is gone.

PersistentVolumes can have various reclaim policies, including “Retain”, “Recycle”, and “Delete”. For dynamically provisioned PersistentVolumes, the default reclaim policy is “Delete”. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume contains precious data. Notice that the RECLAIM POLICY is Delete (default value), which is one of the two reclaim policies, the other one is Retain. (A third policy Recycle has been deprecated). In case of Delete, the PV is deleted automatically when the PVC is removed, and the data on the PVC will also be lost.

In that case, it is more appropriate to use the “Retain” policy. With the “Retain” policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume is not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.

This may also happens too when persistent volume is protected. You should be able to cross verify this:

Command:

$ kubectl describe pvc PVC_NAME | grep Finalizers

Output:

Finalizers:    [kubernetes.io/pvc-protection]

You can fix this by setting finalizers to null using kubectl patch:

$ kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=merge

EDIT:

A PersistentVolume can be mounted on a host in any way supported by the resource provider. Each PV gets its own set of access modes describing that specific PV’s capabilities.

The access modes are:

  • ReadWriteOnce – the volume can be mounted as read-write by a single node
  • ReadOnlyMany – the volume can be mounted read-only by many nodes
  • ReadWriteMany – the volume can be mounted as read-write by many nodes

In the CLI, the access modes are abbreviated to:

  • RWO - ReadWriteOnce
  • ROX - ReadOnlyMany
  • RWX - ReadWriteMany

So if you recreated pod and scheduler put it on different node and your PV has reclaim policy set to ReadWriteOnce it is normal that you cannot access your data.

Claims use the same conventions as volumes when requesting storage with specific access modes. My advice is to edit PV access mode to ReadWriteMany.

$ kubectl edit pv your_pv

You should be updating the access mode in PersistentVolume as shown below

   accessModes:
      - ReadWriteMany
Malgorzata
  • 6,409
  • 1
  • 10
  • 27
  • So the question how do I get the pod to bind back to the original pvc? – Rutnet Oct 17 '19 at 11:26
  • I’m using dynamic block storage on DigitalOcean on a single node. There is currently no capability for ReadWriteMany – Rutnet Oct 17 '19 at 16:12
  • Can you provide your configs file ? – Malgorzata Nov 07 '19 at 12:05
  • Doesn’t the access mode mean only one node can mount *at one time* (rather than ever)? Will the sxheduler place a pod where the PV is unavailable? How is the reclaim policy relevant if the PVC is outliving the pod anyway? – benjimin Oct 20 '22 at 16:26
  • @Malgorzata Do you indeed mean `reclaim policy set to ReadWriteOnce` or rather `access mode set to ReadWriteOnce` ? – mloskot Feb 10 '23 at 17:11