3

Kubernetes version: V1.22.2

Cloud Provider Vsphere version 6.7

Architecture:

  • 3 Masters
  • 15 Workers

What happened: One of the pods for some "unknown" reason went down, and when we try to lift him up, it couldn't attach the existing PVC. This only happened to a specific pod, all the others didn't have any kind of problem.

What did you expect to happen: Pods should dynamically assume PVCs

Validation: First step: The connection to Vsphere has been validated, and we have confirmed that the PVC exists. Second step: The Pod was restarted (Statefulset 1/1 replicas) to see if the pod would rise again and assume the pvc, but without success. Third step: Made a restart to the services (kube-controller, kube-apiserve, etc) Last step: All workers and masters were rebooted but without success, each time the pod was launched it had the same error ""Multi-Attach error for volume "pvc......" Volume is already exclusively attached to one node and can't be attached to another""

When I delete a pod and try to recreate it, I get this warning: Multi-Attach error for volume "pvc-xxxxx" The volume is already exclusively attached to a node and cannot be attached to another

Anything else we need to know: I have a cluster (3 master and 15 nodes)

Temporary resolution: Erase the existing PVC and launch the pod again to recreate the PVC. Since this is data, it is not the best solution to delete the existing PVC.

Multi-Attach error for volume "pvc-xxx" Volume is already exclusively attached to one node and can't be attached to another

ERROR

Ruben Gonçalves
  • 41
  • 1
  • 3

1 Answers1

4

A longer term solution is referring to 2 facts:

  1. You're using ReadWriteOnce access mode where the volume can be mounted as read-write by a single node.

  2. Pods might be schedule by K8S Scheduler on a different node for multiple reason.

Consider switching to ReadWriteMany where the volume can be mounted as read-write by many nodes.

Rot-man
  • 18,045
  • 12
  • 118
  • 124