2

I am facing an OpenEBS issue in my K8s Infrastructure which is deployed on AWS EKS with 3 nodes. I am deploying a statefulset of RabbitMQ with one replica. I want to persist the RabbitMQ pod data when the node goes down and the pod restarts on other node. So, I deployed OpenEBS in my cluster. I tried to terminate the node in which the pod was running, So the pod tried to restart in other node. But the pod did not start in other node and remained in ContainerCreating State and showed me following issue -

Events:
  Type     Reason              Age    From                     Message
  ----     ------              ----   ----                     -------
  Normal   Scheduled           2m28s  default-scheduler        Successfully assigned rabbitmq/rabbitmq-0 to ip-10-0-1-132.ap-south-1.compute.internal
  Warning  FailedAttachVolume  2m28s  attachdetach-controller  Multi-Attach error for volume "pvc-b62d32f1-de60-499a-94f8-3c4d1625353d" Volume is already exclusively attached to one node and can't be attached to another
  Warning  FailedMount         2m26s  kubelet                  MountVolume.SetUp failed for volume "rabbitmq-token-m99tw" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount         25s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[configuration data rabbitmq-token-m99tw]: timed out waiting for the condition

Then after sometime(around 5-10 minutes), the rabbitmq pod was able to start but I observed that one cstor-disk-pool pod is failing with following error -

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  7m7s (x3 over 7m9s)  default-scheduler  0/2 nodes are available: 2 node(s) didn't match node selector.
  Warning  FailedScheduling  44s (x8 over 6m14s)  default-scheduler  0/3 nodes are available: 3 node(s) didn't match node selector.

I described that cstor-disk-pool pod, and the Node-Selectors key still has the value of the old node(which was terminated)Can someone please help me with this issue? Also, We need a way to reduce the time for the rabbitmq pod to restart and get ready properly as we can't afford a downtime of 5-10 minutes of rabbitmq service for our application

Jay Kothari
  • 21
  • 1
  • Can you share your stateful yaml? Is there any node (anti)affinity assigned? Did you drain the node before restarting? – kool Dec 02 '20 at 12:37
  • I am using rabbitmq's helm chart by bitnami for deploying rabbitmq - https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq. I have implemented OpenEBS as storage service for persisting my data on node failures. I was testing a node failure scenario, So I terminated the worker node EC2 instance in which my rabbitmq pod was running from the AWS Console to check if it gets redeployed properly on other node but what i observed is that it takes 5-10 minutes of time to get redeployed on other node and one cstor-disk-pool pod in openebs namespace keeps on failing – Jay Kothari Dec 03 '20 at 13:10
  • Did you modify `values.yaml` from rabbitmq chart? By default `terminationGracePeriodSeconds` is `120 seconds` – kool Dec 04 '20 at 15:59

1 Answers1

1

For the volumes to sustain one node failures, you will need to have created:

  • 3 cStor pools - one on each node
  • Volume should be configured with 3 replicas so data is replicated to all the 3 nodes.

When one of the node is gone, the volume will be able to serve data from the remaining two replicas.

(For making the pod move faster from failed node to new node, you will have to configure the tolerations appropriately. Default is 5 mins).

The cStor pools are tied to the node on which they are created. This is done to allow re-use of the data from the pool when the node comes back. There are a few solutions depending on the way your nodes and disks are configured, that can help you automate the process of running the cstor pools or move them from failed node to a new node. Could you join the Kubernetes slack #openebs channel or create an issue on the openebs github to take further help?

Kiran Mova
  • 113
  • 6