Kubernetes: Can't delete PersistentVolumeClaim (pvc)

Question

I created the following persistent volume by calling

kubectl create -f nameOfTheFileContainingTheFollowingContent.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-monitoring-static-content
spec:
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/some/path"

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-monitoring-static-content-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ""
  resources:
    requests:
      storage: 100Mi

After this I tried to delete the pvc. But this command stuck. when calling kubectl describe pvc pv-monitoring-static-content-claim I get the following result

Name:          pv-monitoring-static-content-claim
Namespace:     default
StorageClass:
Status:        Terminating (lasts 5m)
Volume:        pv-monitoring-static-content
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
Finalizers:    [foregroundDeletion]
Capacity:      100Mi
Access Modes:  RWO
Events:        <none>

And for kubectl describe pv pv-monitoring-static-content

Name:            pv-monitoring-static-content
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
Finalizers:      [kubernetes.io/pv-protection foregroundDeletion]
StorageClass:
Status:          Terminating (lasts 16m)
Claim:           default/pv-monitoring-static-content-claim
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        100Mi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /some/path
    HostPathType:
Events:            <none>

There is no pod running that uses the persistent volume. Could anybody give me a hint why the pvc and the pv are not deleted?

score 231 · Accepted Answer · edited Jun 25 '19 at 21:42

231

This happens when persistent volume is protected. You should be able to cross verify this:

Command:

kubectl describe pvc PVC_NAME | grep Finalizers

Output:

Finalizers: [kubernetes.io/pvc-protection]

You can fix this by setting finalizers to null using kubectl patch:

kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=merge

Ref; Storage Object in Use Protection

edited Jun 25 '19 at 21:42

codersofthedark

9,183
8
45
70

answered May 17 '19 at 08:55

Xiak

2,425
2
10
5

that solution works best than edit solution accross corporate firewalls – jacktrade Sep 27 '19 at 13:15
3

@codersofthedark it does not explain the cause. Of course it is protected. That's what I already mentioned in my question. But the volume wasn't used by any Pod => protection shouldn't have any effect. – Yannic Bürgmann Feb 27 '20 at 06:29
1

I am having this issue and when I tried the command above to patch the pvc, kept getting the error `unable to parse "'{metadata:{finalizers:": yaml: found unexpected end of stream` – phydeauxman Mar 20 '20 at 10:01
for my case, the PVCs are protected cuz I only deleted StatefulSet, not the underlying pods, so the PVCs are still being used by Pods, that's why it is in TERMINATING phase – DiveInto Jun 17 '20 at 01:42
I'm on GKE and something seems to be setting the finalizer back immediately. :/ – weberc2 Mar 05 '21 at 16:49
@weberc2 I have the same issue, have you found any solution? – Ardi Dec 16 '22 at 16:24

score 24 · Answer 2 · answered Jul 17 '18 at 18:19

24

I'm not sure why this happened, but after deleting the finalizers of the pv and the pvc via the kubernetes dashboard, both were deleted. This happened again after repeating the steps I described in my question. Seems like a bug.

answered Jul 17 '18 at 18:19

Yannic Bürgmann

6,301
5
43
77

8

I had a similar problem: PVC didn't want to die and because of that the project was in "Terminating" state forever. I did `oc edit pvc/protected-pvc -n myproject` and deleted those two lines about finalizers. Both PVC and the projects were gone immediately. I agree it's probably a bug because it should not behave that way. I didn't have any pods running in that project, just that PVC. – Pavel Anni Aug 30 '18 at 18:04
1

I just came across this same problem, with this being the solution for me too...delete the constraints. This isn't a "thank you" comment. Rather, I'm adding this because it's 7+ months later and this problem still seems to exist in the wild, and thought new readers might benefit in knowing that. I'm running the latest 'minikube' (installed and built just a few days ago) behind an up to date "Docker for Mac". – CryptoFool Apr 06 '19 at 20:36
...I'm following an online tutorial. I don't know if this is related to this bug, but the behavior I get is different than the instructor's. He creates a new PVC and its state is initially "Pending". Only when he manually creates a PV does the state of the PVC become "bound". In my case, it seems that creating the PVC using the same command he does immediately creates a PV to use the PVC allocated storage. Does anyone know why this is? – CryptoFool Apr 06 '19 at 20:41
1

As answer is not complete in my opinion (not explaining steps of a solution for laics) - You can remove Finalizers in dashboard in YAML of particular PV. Additionally you can do that in terminal by: `kubectl patch pvc NAME -p '{"metadata":{"finalizers":null}}'`, `kubectl patch pod NAME -p '{"metadata":{"finalizers":null}}'`. Source https://github.com/kubernetes/kubernetes/issues/69697#issuecomment-489876217 – Uliysess Jul 25 '19 at 12:33
Already mentioned in another answer: https://stackoverflow.com/a/56182934/2576531 – Yannic Bürgmann Jul 26 '19 at 04:53

Ali Atakan · Answer 3 · 2019-06-05T08:27:37.323

24

You can get rid of editing your pvc! Remove pvc protection.

kubectl edit pvc YOUR_PVC -n NAME_SPACE
Manually edit and put # before this line
All pv and pvc will be deleted

edited Jun 05 '19 at 08:27

answered Jun 05 '19 at 08:17

Ali Atakan

429
4
5

This comment has helped me with deleting a volume that was stuck in Terminating state. thanks. – Rakesh Gupta Jun 05 '19 at 18:17

score 15 · Answer 4 · edited Sep 25 '18 at 21:45

15

The PV is protected. Delete the PV before deleting the PVC. Also, delete any pods/ deployments which are claiming any of the referenced PVCs. For further information do check out Storage Object in Use Protection

edited Sep 25 '18 at 21:45

Leo K

5,189
3
12
27

answered Jul 16 '18 at 11:38

georgekuruvillak

325
2
10

1

I tried to delete both. The pv and the pvc. As you can see in the describe output both are in terminating state – Yannic Bürgmann Jul 16 '18 at 12:14
What platform are you using? Have you tried to delete using `kubectl create -f nameOfTheFileContainingTheFollowingContent.yaml` ? – Vit Jul 17 '18 at 10:31

score 12 · Answer 5 · answered May 19 '20 at 16:11

For me pv was in retain state, hence doing the above steps did not work.

1st we need to change policy state as below :

kubectl patch pv PV_NAME -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'

Then delete pvc as below.

kubectl get pvc

kubectl delete pvc PVC_NAME

finally, delete pv with

kubectl delete pv PV_NAME

score 6 · Answer 6 · answered Oct 18 '18 at 04:07

6

Just met this issue hours ago.

I deleted deployments that used this references and the PV/PVCs are automatically terminated.

answered Oct 18 '18 at 04:07

congpham

71
1
3

Thank you! this works for me – haruhi Sep 22 '21 at 19:56

score 5 · Answer 7 · answered Nov 14 '18 at 17:14

If PV still exists it may be because it has ReclaimPolicy set to Retain in which case it won't be deleted even if PVC is gone. From the docs:

PersistentVolumes can have various reclaim policies, including “Retain”, “Recycle”, and “Delete”. For dynamically provisioned PersistentVolumes, the default reclaim policy is “Delete”. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume contains precious data. In that case, it is more appropriate to use the “Retain” policy. With the “Retain” policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume is not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered

score 5 · Answer 8 · answered Mar 09 '19 at 06:39

5

In my case, as long as I delete the pod associated to both pv and pvc, the pv and pvc in terminating status are gone

answered Mar 09 '19 at 06:39

j3ffyang

1,049
12
12

"There is no pod running that uses the persistent volume" – Yannic Bürgmann Mar 10 '19 at 04:41
I met this issue again today. 2 PVs, without Pod and PVC associated, turned into ```terminating``` state forever, when being deleted. To fix it, I ran ```kubectl patch pv local-pv-324352d9 -n ops -p '{"metadata":{"finalizers": []}}' --type=merge``` Then the PV is gone. Thanks @Xiak hint – j3ffyang Jun 01 '19 at 02:26

score 0 · Answer 9 · answered Nov 14 '19 at 11:14

0

in my case a pvc was not deleted because missing namespace (I deleted the namespace before deleting all resources/pvc) solution : create namespace with the same name as it was before and then I was able to remove the finalizers and finally pvc

answered Nov 14 '19 at 11:14

dansl1982

998
1
7
10

score 0 · Answer 10 · answered Dec 17 '20 at 11:16

In case you have already deleted PV and trying to delete PVC

Check if the volume is attached by this command

kubectl get volumeattachment

Deleting the PVC :-

First you have to delete pvc pne by one using this command

kubectl delete pvc <pvc_name> --grace-period=0 --force

Or you can delete all PVC's using

kubectl delete pvc --all

Now you can see the status of PVC as terminating by using

kubectl get pvc

and then you have to apply this delete using

kubectl patch pvc {PVC_NAME} -p '{"metadata":{"finalizers":null}}'

score 0 · Answer 11 · edited Jun 15 '21 at 00:17

0

kubectl get pvc pvc_name -o yaml > pvcfile.yaml

Then open pvcfile.yaml and delete the finalizers line, save and apply :

kubectl apply -f pvcfile.yaml

edited Jun 15 '21 at 00:17

DaveL17

1,673
7
24
38

answered Jun 10 '21 at 16:02

wiem lim

1

1

Instead of two commands, it can be done as 'kubectl edit pvc pvc_name', then remove the section and save. BTW, as per the original question, the author removed the PVC - but that operation failed due to other reasons as specified in the above answers. – Rajan Panneer Selvam Jun 10 '21 at 18:29

Kamafeather · Answer 12 · 2022-02-16T16:27:28.780

Additionally to the other answers about the finalizer...

I could free up the resources only after deleting the Deployment. After that, the Terminating resources got released.

Delete all the resources listed by:

kubectl get all -n YOURNAMESPACE

Use kubectl delete -n YOURNAMESPACE <resource> <id> or (if you copy paste from the above output) kubectl delete -n YOURNAMESPACE <resource>/<id>, for each resource that you see listed there.

You can also do it at once

kubectl delete -n YOURNAMESPACE <resource>/<id1>  <resource>/<id2>  <resource2>/<id3>  <resource2>/<id4>  <resource3>/<id5>

Probably you tried to remove resources but they are getting recreated because of the deployment or replicaset resource, preventing the namespace from freeing up depending resources and from being cleaned up.

This is the expected behaviour for protected volumes. When the deployment still uses the volume it cant be deleted. — Yannic Bürgmann, Feb 15 '22 at 21:32

Kubernetes: Can't delete PersistentVolumeClaim (pvc)

12 Answers12

Linked