Can 3 replicas use the same PersistentVolume in a StatefulSet in Kubernetes?

Question

I created a StatefulSet for running my NodeJS with 3 replicas and want to attach to a gce disk that can become a data storage for user to upload files.

My project naming: carx; Server name: car-server

However I got an error while creating the second pod.

kubectl describe pod car-server-statefulset-1

AttachVolume.Attach failed for volume "my-app-data" : googleapi: Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE - The disk resource 'projects/.../disks/carx-disk' is already being used by 'projects/.../instances/gke-cluster-...-2dw1'

car-server-statefulset.yml

apiVersion: v1
kind: Service
metadata:
  name: car-server-service
  labels:
    app: car-server
spec:
  ports:
  - port: 8080
    name: car-server
  clusterIP: None
  selector:
    app: car-server
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: car-server-statefulset
spec:
  serviceName: "car-server-service"
  replicas: 3
  template:
    metadata:
      labels:
        app: car-server
    spec:
      containers:
        - name: car-server
          image: myimage:latest
          ports:
            - containerPort: 8080
              name: nodejs-port
          volumeMounts:
          - name: my-app-data
            mountPath: /usr/src/app/mydata
      volumes:
      - name: my-app-data
        persistentVolumeClaim:
          claimName: example-local-claim
  selector:
    matchLabels:
      app: car-server

pvc.yml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-local-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: standard

pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-app-data
  labels:
    app: my-app
spec:
  capacity:
    storage: 60Gi
  storageClassName: standard
  accessModes:
    - ReadWriteMany
  gcePersistentDisk:
    pdName: carx-disk
    fsType: ext4

Jonas · Answer 1 · 2020-09-23T16:46:34.543

4

The Access Mode field is treated as a request, but it is not sure that you get what you requests. In your case, GCEPersistentDisk only support ReadWriteOnce or ReadOnlyMany.

Your PV is now mounted as ReadWriteOnce but can only be mounted on one node at the same time. So the other replicas will fail to mount the volume.

When using StatefulSet, it is common that each replica use its own volume, use the volumeClaimTemplate: part of the StatefulSet manifest for that.

Example:

  volumeClaimTemplates:
  - metadata:
      name: example-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 5Gi

In case that you only can use a single volume, you may consider to run the StatefulSet with only one replica, e.g. replicas: 1.

If you want disk-replication, you can use a StorageClass for regional disks that are replicated to another AZ as well. See Regional Persistent Disk, but it still have the same access modes.

edited Sep 23 '20 at 16:46

answered Sep 23 '20 at 16:38

Jonas

121,568
97
310
388

Thanks for reply! if each replica use its own volume, the data file in those volumes will not be synchronized automatically? I required set another configuration to do that? – potato Sep 23 '20 at 16:40
@potato, yes that is true. Use only 1 replica if your app does not replicate the data. – Jonas Sep 23 '20 at 16:41
So that's means I cannot configure 3 replicas to share the data file on one PV in GCE? because I want my app being scalable. So is this a right way to make each replica use its own volume and synchronize data between volumes for a scalable app? – potato Sep 23 '20 at 16:49
@potato it is much easier to do the server scalable if it does not use a volume, e.g. use a `Deployment` without volume. Maybe you can use a GCS bucket or a database for the things that you want to use a volume for instead? – Jonas Sep 23 '20 at 16:51
My app is going to allow users to upload files with 1MB each, so I would like to use disk storage like storing images. I will look over GCS bucket whether is suitable for me. Well thanks for the suggestions. – potato Sep 23 '20 at 17:01

score 1 · Answer 2 · answered Sep 23 '20 at 16:49

From the docs

A feature of PD is that they can be mounted as read-only by multiple consumers simultaneously. This means that you can pre-populate a PD with your dataset and then serve it in parallel from as many Pods as you need. Unfortunately, PDs can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed.

Alternative solution is Google Cloud Filestore which is a NAS offering. You can mount the Filestore in Compute Engine and Kubernetes Engine instances. However, the problem with Filestore is that it designed with large file storage systems in mind and has minimum capacity of 1TB which is expensive for small use cases.

An inexpensive way to solve the issue is to setup a NFS server in your cluster backed up by a ReadWriteOnce PV and then create NFS based PV (which supports ReadWriteMany) using this NFS server

Can 3 replicas use the same PersistentVolume in a StatefulSet in Kubernetes?

2 Answers2