How to reattach released PersistentVolume in Kubernetes

Question

Here is my overall goal:

• Have a MongoDB running

• Persist the data through pod failures / updates etc

The approach I’ve taken:

• K8S Provider: Digital Ocean

• Nodes: 3

• Create a PVC

• Create a headless Service

• Create a StatefulSet

Here’s a dumbed down version of the config:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: some-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: do-block-storage
---
apiVersion: v1
kind: Service
metadata:
  name: some-headless-service
  labels:
    app: my-app
spec:
  ports:
  - port: 27017
    name: my-app-database
  clusterIP: None
  selector:
    app: my-app
    tier: database
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-app-database
  labels:
    app: my-app
    tier: database
spec:
  serviceName: some-headless-service
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      tier: database
  template:
    metadata:
      labels:
        app: my-app
        tier: database
    spec:
      containers:
      - name: my-app-database
        image: mongo:latest
        volumeMounts:
        - name: some-volume
          mountPath: /data
        ports:
        - containerPort: 27017
          name: my-app-database
      volumes:
      - name: some-volume
        persistentVolumeClaim:
          claimName: some-pvc

This is working as expected. I can spin down the replicas to 0:

kubectl scale —replicas=0 statefulset/my-app-database

Spin it back up:

kubectl scale —replicas=1 statefulset/my-app-database

And the data will persist..

But one time, as I was messing around by scaling the statefulset up and down, I was met with this error:

Volume is already exclusively attached to one node and can't be attached to another

Being new to k8s, I deleted the PVC and “recreated” the same one:

kubectl delete pvc some-pvc
kubectl apply -f persistent-volume-claims/

The statefulset spun back up with a new PV and the old PV was deleted as the persistentVolumeReclaimPolicy was set to Delete by default.

I set this new PV persistentVolumeReclaimPolicy to Retain to ensure that the data would not be automatically removed.. and I realized: I’m not sure how I’d reclaim that PV. Earlier to get through the “volume attachment” error, I deleted the PVC, which will just create another new PV with the setup I have, and now I’m left with my data in that Released PV.

My main questions are:

• Does this overall sound like the right approach for my goal?

• Should I look into adding a claimRef to the dynamically created PV and then recreating a new PVC with that claimRef, as mentioned here: Can a PVC be bound to a specific PV?

• Should I be trying to get that fresh statefulset PVC to actually use that old PV?

• Would it make sense to try to reattach the old PV to the correct node, and how would I do that?

Answer 1

If your want to use StatefulSet with scalability, your storage should also support this, there are two way to handle this:

• If do-block-storage storage class supprt ReadWriteMany, then put all pod's data in single volume.

• Each pod use a different volume, add volumeClaimTemplate to your StatefulSet.spec, then k8s will create PVC like some-pvc-{statefulset_name}-{idx} automatically:

spec:
  volumeClaimTemplates:
  - metadata:
      name: some-pvc
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: do-block-storage

Update:

StatefulSet replicas Must deploy with mongodb replication, then each pod in StatefulSet will has same data storage.

So when container run mongod command, you must add option --replSet={name}. when all pods up, execute command rs.initiate() to tell mongodb how to handle data replication. When you scale up or down StatefulSet, execute command rs.add() or rs.remove() to tell mongodb members has changed.