Kubernetes volume duplication between worker nodes

Question

I have a simple web app that uses volume/persistent volume claim to serve static data from there. Pods got scheduled only on the first worker node where volume resides. How to deal with the shared volume between nodes in case I want to scale pods and have them allocated across all the available nodes?

      - name: app-nginx
        image: nginx:1.20.0
        command: ["/usr/sbin/nginx", "-g", "daemon off;"]
        volumeMounts:
        - name: static-volume
          mountPath: "/opt/static"

      volumes:
      - name: static-volume
        persistentVolumeClaim:
          claimName: pvc-static

One option is to use NFS and not physically allocate volume on an EC2 instance. Another way is to duplicate static files for each pod (populate them into proper directory with init container) but that requires extra time and is not feasible for a lot a static data.

What's the proper way to deal with such a case within Kubernetes? Is there a way to declare deployment which will be using logically same volume but physically different instances located on the different nodes?

Seems like this one https://stackoverflow.com/questions/51212904/kubernetes-pvc-with-readwritemany-on-aws is the option I need to go with — Most Wanted, Jul 07 '21 at 08:38

score 4 · Accepted Answer · answered Jul 07 '21 at 11:19

What you are looking for is a volume provider that supports the ReadOnlyMany or ReadWriteMany Access Mode.

Follow the documentation link to get a list of the officially supported ones.

If you are on AWS than probably using EFS through the NFS plugin will be the easiest solution, but please take into account the fact it is an NFS-based solution and you might hit a performance penalty.

As a side note, what you are trying to do smeels like an anti-pattern to me. Docker images for an application should be self contained. In your case, having a container serving a static website should contain all the static files it needs to in order to be fully portable. This would remove the need to have an external volume with the data completely.

Harsh Manvar · Answer 2 · 2021-07-07T13:07:20.040

Yes, you are right one option is to use the NFS. You have to implement the ReadWriteMany or ReadOnlyMany : https://stackoverflow.com/a/57798369/5525824

If you have a scenario of ReadOnlyMany you can create the PVC in GCP with GKE. https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/readonlymany-disks

However, if you are looking forward to doing a write operation also the available option is to use FileSystem or NFS.

You can also checkout implementing the Minio if not want to use any managed service and following cloud-agnostic : https://min.io/

NAS : https://github.com/minio/charts#nas-gateway

Just for FYI : https://github.com/ctrox/csi-s3 performance might be not good.

Kubernetes volume duplication between worker nodes

2 Answers2