2

I am using Kustomize to manage my Kubernetes project with a StetefulSet that deploys a PostgreSQL cluster with three pods. I am working on Vagrant/VirtualBox so no dynamic provisioning of PV exists. For this reason, I have my own pv.yaml containing the manifest to deploy these 3 PVs.

Then I have a kustomization.yaml file like this:

namespace: ibm-cfdb

bases:
- ../../bases

resources:
- pv.yaml

the folder ../../bases contains the file to deploy the StatefulSet. When I run: kubectl apply -k kustomize/ everything is correctly deployed. PVs are created before the StetefulSet that contains a volumeClaimTemplates that declare the Claim for these PVs.

The problem is that when I try to remove the deployment with the command: kubectl delete -k kustomize/ the removal of PV is executed (it seems I don't have control about the order). I suppose these PVs cannot be deleted because Claims use them. Then the StatefulSet removal stuck.

What is the best approach to manage PV static provisioning with Kustomize?

Salvatore D'angelo
  • 1,019
  • 3
  • 14
  • 39
  • After a bit of investigation, I found that the Stateful removal doesn't remove PVCs and I understand it. The problem is that Kustomize should not call ```pv.yaml``` on delete. Is there a way to achieve this? – Salvatore D'angelo Aug 06 '21 at 15:19
  • Avoid executing `kubectl delete` for directories that contain PVCs if you don't want to delete them. – Jonas Aug 06 '21 at 15:26
  • PVCs are not deleted. PVs are. To avoid issues I need to move PV to another folder but this adds another command to remember when I deploy the app. I hoped a feature like "ignore pv.yaml on delete" but I don't think it exists. – Salvatore D'angelo Aug 06 '21 at 15:32
  • Try to use a dynamic volume provisioner, if available in your system - that way you don't have to create PVs yourself. – Jonas Aug 06 '21 at 15:34
  • I know, my app deploys on Vagrant/VirtualBox where I use static provisioning and then the ```pv.yaml``` file. On IBM Cloud I use the dynamic provision and this issue doesn't exist. – Salvatore D'angelo Aug 06 '21 at 17:50

1 Answers1

1

You encountered an interesting problem regarding StatefulSet and PVC removal. There is a discussion whether PVCs created by the StatefulSet should be deleted when deleting the corresponding StatefulSet. We recently received information that the feature to autodelete the PVCs created by StatefulSet will probably be available in the 1.23 release. According to the feature documentation, this will allow us to specifiy if the VolumeClaimTemplate PVCs will be deleted after deleting their StatefulSet. I suspect that with this feature it'll be easy to delete your StatefulSet along with PVC and PV.

For now, you can consider moving the file with the PV to another directory and manage it separately. However, I will propose another solution which is kind of a workaround but you may be interested.

Basically, we can use the -o flag with the kustomize build command. This creates one file per resource, which gives us more control over resources creation.

I will give you an example to illustrate how this can work.

Suppose I have a similar environment to you:

$ tree
.
├── base
│   ├── kustomization.yaml
│   └── statefulset.yaml
└── overlays
    └── dev
        ├── kustomization.yaml
        └── pv.yaml

$ cat overlays/dev/kustomization.yaml
bases:
  - ../../base

resources:
- pv.yaml

Now let's create a directory where our manifest files generated by kustomize will be stored:

$ mkdir generated_manifests

Then we can check if the command kustomize build overlays/dev -o generated_manifests works as expected. First we'll apply the generated manifests (it'll create the web StatefulSet and pv0003 PersistentVolume):

$ kustomize build overlays/dev -o generated_manifests && kubectl apply -Rf generated_manifests/
statefulset.apps/web created
persistentvolume/pv0003 created

As you can see, the appropriate manifest files have been created in the generated_manifests directory:

$ ls generated_manifests/
apps_v1_statefulset_web.yaml  v1_persistentvolume_pv0003.yaml

Finally, we can try to delete only the web StatefulSet:

$ kustomize build overlays/dev -o generated_manifests && kubectl delete -f generated_manifests/apps_v1_statefulset_web.yaml
statefulset.apps "web" deleted

I would also like to mention that kustomize has a feature like "ignore pv.yaml" but it will also be used when creating resources, not just when removing. This is known as a delete patch and a good example can be found here.

matt_j
  • 4,010
  • 1
  • 9
  • 23
  • Just read your solution. To be honest I simply separate the StatefulSet resources from PV resources and called them in separated files. When I remove them I first remove the StatefulSet, then the PVC, then the PV. I will analyze your proposal with more attention. For the moment I upvoted it. – Salvatore D'angelo Aug 31 '21 at 16:12