delete Kubernetes persistent volume from statefulset after scale down

Question

I scaled my statefulset up to 4, and when scaling down to 1, I saw that I still have 4 persistent volumes with indexes from 0 to 3.

I also saw that the status of all of them is Bound I guess it is because I use it as stateful set, so it doesn't delete the volumes after scale down.

I tried to manually delete one of tham (the one with index 2) because I was sure it will release my volume, so I used:

kubectl delete persistentvolume <volume>

Well, that didn't help, it just made this volume to be in a terminating state forever... :/

I have no idea how to remove this and all the other unused volumes now.

here is the volume configuration in stateful set yaml.

  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "default"
        resources:
          requests:
            storage: 7Gi

if I run

kubectl get pvc --all-namespaces

I get

NAMESPACE    NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default      data-0         Bound    pvc-23af1aec-e385-4778-b0b0-56f1d1dfdfee   7Gi        RWO            default        4h5m
default      data-1         Bound    pvc-34625107-1352-4715-b12c-2fc6ff22ed08   7Gi        RWO            default        4h4m
default      data-2         Bound    pvc-15dbdb53-d951-465d-b9c3-ebadfcc3c725   7Gi        RWO            default        4h3m
default      data-3         Bound    pvc-d317657f-194a-4f4f-8c5f-dff2843b693f   7Gi        RWO            default        4h3m

if I run

kubectl get --no-headers persistentvolumes

I get this:

pvc-15dbdb53-d951-465d-b9c3-ebadfcc3c725   7Gi   RWO   Delete   Terminating   default/data-2            default         4h4m
pvc-23af1aec-e385-4778-b0b0-56f1d1dfdfee   7Gi   RWO   Delete   Bound         default/data-0            default         4h6m
pvc-34625107-1352-4715-b12c-2fc6ff22ed08   7Gi   RWO   Delete   Bound         default/data-1            default         4h5m
pvc-d317657f-194a-4f4f-8c5f-dff2843b693f   7Gi   RWO   Delete   Bound         default/data-3            default         4h3m

can you share the output `kubectl get pvc --all-namespaces` for this? — Emon46, Jan 12 '21 at 19:10
can you please run the deletation command for pvc delete ? i have given the command in my answer — Emon46, Jan 12 '21 at 19:29

Emon46 · Accepted Answer · 2021-01-13T06:39:26.197

7

In statefulset, K8s doesn't delete PV or PVC by their own after termination of a pod automatically, It is to avoid further complication and for data safety. Thats why after doing scale down, we need to do it manually.Deleting the PVC after the pods have terminated will trigger deletion of the respective Persistent Volumes depending on the storage class and reclaim policy.

Please try to delete persistent volume claim or PVC instead of persistent volume. if you delete pvc it will automatically delete the respective pv.

just run this command in your bash:

kubectl delete pvc data-3

REF

edited Jan 13 '21 at 06:39

answered Jan 12 '21 at 19:27

Emon46

1,506
7
14

1

I believe the outcome of deleting the PVC may also depend on whether the PV's `persistentVolumeReclaimPolicy` is set to `retain` or `delete`. For more information, see https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/ – rolfedh Mar 05 '21 at 02:45

Shashi Deshetti · Answer 2 · 2022-06-20T11:53:51.103

2

You can set the retention policy to define the behavior for whenDeleted and whenScaled.

Check the same here:

edited Jun 20 '22 at 11:53

answered Jun 20 '22 at 11:43

Shashi Deshetti

1,354
2
11
22

score 2 · Answer 3 · answered Sep 06 '22 at 11:55

There is a new feature introduced in k8s v1.23 [alpha] for this exact requirement. Look for PVC retention policy in STS.

whenDeleted

configures the volume retention behavior that applies when the StatefulSet is deleted

whenScaled

configures the volume retention behavior that applies when the replica count of the StatefulSet is reduced; for example, when scaling down the set.

For each policy that you can configure, you can set the value to either Delete or Retain.

Something like below would work for you.

On STS deletion - PVC, PV, cloud volume will be retained.
On STS scale down - PVC, PV, cloud volume will be deleted.

below snippet from k8s docs:

apiVersion: apps/v1
kind: StatefulSet
...
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete
...

Along with this, Don't forget to set the PV reclaim policy to delete. This will ensure that as soon as PVC gets deleted, even the PV along with backing volume would get deleted.

refer this diagram for better understanding this.

delete Kubernetes persistent volume from statefulset after scale down

3 Answers3