Exporting PersistentVolumes and PersistentVolumesClaims from Kubernetes API

Question

On GKE I created a statefulset containing a volumeClaimTemplates. Then all the related PersistentVolumesClaims, PersistentVolumes and Google Persistent Disks are automatically created:

kubectl get pvc
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
        76m
qserv-data-qserv-worker-0      Bound    pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-1      Bound    pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-2      Bound    pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-3      Bound    pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-4      Bound    pvc-7c49e7a0-fd73-467d-b677-820d899f41ee   3Gi        RWO            qserv          76m

kubectl get pv
pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-2      qserv                   77m
pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-1      qserv                   77m
pvc-7c49e7a0-fd73-467d-b677-820d899f41ee   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-4      qserv                   77m
pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-3      qserv                   77m
pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-0      qserv                   77m

gcloud compute disks list                                                                                                                   
NAME                                           LOCATION       LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
...
pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79       us-central1-c  zone            3        pd-balanced  READY
pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f       us-central1-c  zone            3        pd-balanced  READY
pvc-7c49e7a0-fd73-467d-b677-820d899f41ee       us-central1-c  zone            3        pd-balanced  READY
pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07       us-central1-c  zone            3        pd-balanced  READY
pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b       us-central1-c  zone            3        pd-balanced  READY

Is there a simple way to extract PVC/PV yaml file so that I can re-create all PVs/PVCs using the same Google Disks. (This might be useful to move the data to a new GKE cluster in case I delete the current one, or to restore the data if somebody remove accidentally the PVCs/PVs)

kubectl get pv,pvc -o yaml > export.yaml

Above command does not work because there is too much technical fields set at runtime which prevent kubectl apply -f export.yaml to work. Would you know a way to remove these fields from export.yaml

Answer 1

As asked in the question:

Is there a simple way to extract PVC/PV yaml file so that I can re-create all PVs/PVCs using the same Google Disks.

Some scripting would be needed to extract a manifest that could be used without any hassles.

I found a StackOverflow thread about similar question (how to export manifests):

• Stackoverflow.com: Questions: Kubectl export is deprecated any alternative

A side note!

I also stumbled upon kubectl neat (a plugin for kubectl) which will be referenced later in that answer.

As correctly pointed by the author of the post, kubectl neat will show the message at the time of installation:

WARNING: You installed plugin "neat" from the krew-index plugin repository.

These plugins are not audited for security by the Krew maintainers. Run them at your own risk.

I would consider going with some backup solution as more a viable option due to the fact of data persistence and in general data protection in case of any failure.

From backup solution side you can look here:

• Portwortx.com: How to migrate stateful application from one gcp region to another with portwortx kubemotion
• Velero.io

PV's in GKE are in fact Google Persistent Disks. You can create a snapshot/image of a disk as a backup measure. You can also use this feature to test how your migration behaves:

• Cloud.google.com: Compute: Docs: Disks: Create snapshots
• Cloud.google.com: Compute: Docs: Images: Create delete deprecate private images

---

Please consider below example as a workaround.

I've managed to migrate an example Statefulset from one cluster to another with data stored on gce-pd.

Once more, I encourage you to check this docs about using preexisting disks in GKE to create a Statefulset:

• Cloud.google.com: Kubernetes Engine: Docs: How to: Persistent Volumes: Preexisting PD

Assuming that you used manifest from official Kubernetes site:

• Kubernetes.io: Docs: Concepts: Workloads: Controllers: Statefulset

You can migrate it to another cluster by:

• Setting the ReclaimPolicy to Retain on each PV used. <-- IMPORTANT
• Using kubectl neat to extract needed manifests
• Editing previously extracted manifests
• Deleting the existing workload on old cluster
• Creating a workload on new cluster

Setting the ReclaimPolicy to Retain on each PV used

You will need to check if your PV ReclaimPolicy is set to Retain. This would stop gce-pd deletion after PVC and PV are deleted from the cluster. You can do it by following Kubernetes documentation:

• Kubernetes.io: Docs: Tasks: Administer cluster: Change PV reclaim policy

More reference:

• Cloud.google.com: Kubernetes Engine: Docs: Concepts: Persistent Volumes: Dynamic provisioning

Using kubectl neat to extract needed manifests

There are many ways to extract the manifest from Kubernetes API. I stumbled upon kubectl neat here. kubectl-neat will remove some of the fields in the manifests.

I used it in a following manner:

• $ kubectl get statefulset -o yaml | kubectl neat > final-sts.yaml
• $ kubectl get pvc -o yaml | kubectl neat > final-pvc.yaml
• $ kubectl get pv -o yaml | kubectl neat > final-pv.yaml

Disclaimer!

This workaround will use the names of the dynamically created disks in GCP. If you were to create new disks (from snapshot for example) you would need to modify whole setup (use preexsiting disks guide referenced earlier).

Above commands would store manifests of a StatefulSet used in the Kubernetes examples.

Editing previously extracted manifests

You will need to edit this manifests to be used in newly created cluster. This part could be automated:

• final-pv.yaml - delete the .claimRef in .spec

Deleting the existing workload on old cluster

You will need to release used disks so that the new cluster could use them. You will need to delete this Statefulset and accompanying PVC's and PV's. Please make sure that the PV's reclaimPolicy is set to Retain.

Creating a workload on new cluster

You will need to use previously created manifest and apply them on a new cluster:

• $ kubectl apply -f final-pv.yaml
• $ kubectl apply -f final-pvc.yaml
• $ kubectl apply -f final-sts.yaml

---

As for exporting manifests you could also look (if feasible) on Kubernetes client libraries:

• Kubernetes.io: Docs: Reference: Using API: Client libraries

Answer 2

Here is an example script which replace kubectl neat and manual edit of the manifest files (removal of .spec.claimRef field) in order to export the mapping between PVCs,PVs and Google Persistent Disks.

https://github.com/k8s-school/skateful/blob/stackoverflow/main.go

How to use it:

git clone https://github.com/k8s-school/skateful.git
git checkout stackoverflow
# Requirements: go >= 1.14.7 and a kubeconfig file
make
./skateful

it will create a pvc-pv.yaml file that can be applied to any new GKE kubernetes cluster in order to attach the existing Google Persistent Disks to new PVCs/PVs.