Rabbit mq - Error while waiting for Mnesia tables

Question

I have installed rabbitmq using helm chart on a kubernetes cluster. The rabbitmq pod keeps restarting. On inspecting the pod logs I get the below error

2020-02-26 04:42:31.582 [warning] <0.314.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_durable_queue]}
2020-02-26 04:42:31.582 [info] <0.314.0> Waiting for Mnesia tables for 30000 ms, 6 retries left

When I try to do kubectl describe pod I get this error

Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-rabbitmq-0
    ReadOnly:   false
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-config
    Optional:  false
  healthchecks:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-healthchecks
    Optional:  false
  rabbitmq-token-w74kb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rabbitmq-token-w74kb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/arch=amd64
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                      From                                               Message
  ----     ------     ----                     ----                                               -------
  Warning  Unhealthy  3m27s (x878 over 7h21m)  kubelet, gke-analytics-default-pool-918f5943-w0t0  Readiness probe failed: Timeout: 70 seconds ...
Checking health of node rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local ...
Status of node rabbit@rabbitmq-0.rabbitmq-headless.default.svc.cluster.local ...
Error:
{:aborted, {:no_exists, [:rabbit_vhost, [{{:vhost, :"$1", :_, :_}, [], [:"$1"]}]]}}
Error:
{:aborted, {:no_exists, [:rabbit_vhost, [{{:vhost, :"$1", :_, :_}, [], [:"$1"]}]]}}

I have provisioned the above on Google Cloud on a kubernetes cluster. I am not sure during what specific situation it started failing. I had to restart the pod and since then it has been failing.

What is the issue here ?

Have you tried to describe the running pod? Could you provide more information about your setup? Is it cloud provisioned? Is it failing on specific terms or just fails after the `helm install`? — Dawid Kruk, Feb 26 '20 at 10:07
This is the error I get. I have updated the question with the error details ```Error: {:aborted, {:no_exists, [:rabbit_vhost, [{{:vhost, :"$1", :_, :_}, [], [:"$1"]}]]}} Error: {:aborted, {:no_exists, [:rabbit_vhost, [{{:vhost, :"$1", :_, :_}, [], [:"$1"]}]]}} ``` — jeril, Feb 26 '20 at 11:19
ok let me test with the latest helm chart and try once again — jeril, Feb 26 '20 at 12:29
This is the helm chart that I used. https://github.com/helm/charts/tree/master/stable/rabbitmq. These are values that I used - https://github.com/helm/charts/blob/master/stable/rabbitmq/values-production.yaml This part alone was commented in values-production.yaml # extraPlugins: "rabbitmq_auth_backend_ldap — jeril, Feb 26 '20 at 12:43
When I uninstalled and installed rabbitmq using helm it was using the same persistent volume. I tried deleting the persistent volume and reinstalled rabbitmq. The pods are running now without any error. Thanks for the help — jeril, Feb 26 '20 at 12:54

Ulli · Answer 1 · 2021-03-10T15:05:32.910

TLDR

helm upgrade rabbitmq --set clustering.forceBoot=true

Problem

The problem happens for the following reason:

All RMQ pods are terminated at the same time due to some reason (maybe because you explicitly set the StatefulSet replicas to 0, or something else)
One of them is the last one to stop (maybe just a tiny bit after the others). It stores this condition ("I'm standalone now") in its filesystem, which in k8s is the PersistentVolume(Claim). Let's say this pod is rabbitmq-1.
When you spin the StatefulSet back up, the pod rabbitmq-0 is always the first to start (see here).
During startup, pod rabbitmq-0 first checks whether it's supposed to run standalone. But as far as it can see on its own filesystem, it's part of a cluster. So it checks for its peers and doesn't find any. This results in a startup failure by default.
rabbitmq-0 thus never becomes ready.
rabbitmq-1 is never starting because that's how StatefulSets are deployed - one after another. If it were to start, it would start successfully because it sees that it can run standalone as well.

So in the end, it's a bit of a mismatch between how RabbitMQ and StatefulSets work. RMQ says: "if everything goes down, just start everything and the same time, one will be able to start and as soon as this one is up, the others can rejoin the cluster." k8s StatefulSets say: "starting everything all at once is not possible, we'll start with the 0".

Solution

To fix this, there is a force_boot command for rabbitmqctl which basically tells an instance to start standalone if it doesn't find any peers. How you can use this from Kubernetes depends on the Helm chart and container you're using. In the Bitnami Chart, which uses the Bitnami Docker image, there is a value clustering.forceBoot = true, which translates to an env variable RABBITMQ_FORCE_BOOT = yes in the container, which will then issue the above command for you.

But looking at the problem, you can also see why deleting PVCs will work (other answer). The pods will just all "forget" that they were part of a RMQ cluster the last time around, and happily start. I would prefer the above solution though, as no data is being lost.

Also, there is a section about recovery at the Bitnami github, which mentions both forceBoot and one more option (using Parallel podManagementPolicy) https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq#recover-the-cluster-from-complete-shutdown — victorm1710, Jun 01 '21 at 16:51
(1.) FYI the same issue "Error while waiting for Mnesia tables" happens when you have 3 machines running rabbitmq and configured for clustering. so it's not just a kubernetes, helm issue. (2.) unfortunately for rabbitmq there is no config file setting for clustering.forceBoot. so i will have to clear out the /var/lib/rabbitmq/mnesia to get the servers to start. all of them right now are stuck in a boot loop... "inconsistent_database". (3.) in my case if you stop your rabbit cluster with server1,2,3 then then you MUST start in LIFO order i.e. 3,2,1. this allows you to start without issue. — Trevor Boyd Smith, Jun 18 '21 at 20:08
relevant rabbitmq docs: https://www.rabbitmq.com/clustering.html#restarting — Trevor Boyd Smith, Jun 18 '21 at 20:24
This should be marked as answer. Thank you Ulli, you saved me hours of troubleshooting! — Andrey, Sep 21 '21 at 02:14
I had to add `podManagementPolicy=Parallel` as well. Even with the `forceBoot` option `rabbitmq-0` was not starting — dev-rowbot, Jan 21 '22 at 06:30
I am right now in a situation in single node docker swarm (where ordering isn't enforced by statefulset). I started all three rabbitmq nodes (3 services, not 1 with 2 replicas) at the same time after docker upgrade and they are all waiting on each other. Somehow the shutdown caused by containerd restart caused them to think that none of them is the master now and they refuse to elect one. I will set rabbit1 forceBoot=true and see what happens. (Related issue here: https://github.com/helm/charts/issues/13485) — Dalibor Filus, Mar 24 '22 at 17:14

jeril · Answer 2 · 2020-02-27T04:20:51.043

14

Just deleted the existing persistent volume claim and reinstalled rabbitmq and it started working.

So every time after installing rabbitmq on a kubernetes cluster and if I scale down the pods to 0 and when I scale up the pods at a later time I get the same error. I also tried deleting the Persistent Volume Claim without uninstalling the rabbitmq helm chart but still the same error.

So it seems each time I scale down the cluster to 0, I need to uninstall the rabbitmq helm chart, delete the corresponding Persistent Volume Claims and install the rabbitmq helm chart each time to make it working.

edited Feb 27 '20 at 04:20

answered Feb 26 '20 at 12:56

jeril

1,109
2
17
35

2

Be aware that deleting persistent volume claims may destroy your data. Wouldn't do this in production. – Ulli Mar 10 '21 at 15:01
i agree that generally deleting persistent volumes can make you lose your data. but specifically for rabbitmq developers (and many other rabbitmq users i know) we don't need or want or use any rabbitmq persistance features. my colleague informed me that he sets the helm `persistance` to false and doesn't have any `pvc` for rabbitmq. which is what i'll try too. – Trevor Boyd Smith Apr 28 '21 at 13:54
your solution of deleting the pod and pvc worked for me. i think rabbitmq was not shutdown gracefully and that resulted in the mnesia files getting corrupted and so by deleting the pvc and pod i got a new pod with an empty pvc and so the helm chart for rabbitmq was able to finally come back online successfully. – Trevor Boyd Smith Apr 28 '21 at 14:07
1

running rabbitmq helm chart with persistance set to `false` definitely is better: 1. more simple config 2. less resources requested from kubernetes 3. better robustness to coming up and down 4. more reliable (no need for me to write a custom script to check for bad state and then deleting pod/pvc) – Trevor Boyd Smith Apr 28 '21 at 15:19
For anyone else here else experiencing the same issue in tandem with Docker. I experienced this as well and saw this answer.Deleting the RabbitMQ Docker container and having it re-pull the image fixed the issue for our application. – dabyland May 16 '22 at 15:12

score 3 · Answer 3 · answered Mar 21 '22 at 21:54

IF you are in the same scenario like me and you don't know who deployed the helm chart and how was it deployed... you can edit the statefulset directly to avoid messing up more things..

I was able to make it work without deleting the helm_chart

kubectl -n rabbitmq edit statefulsets.apps rabbitmq

under the spec section I added as following the env variable RABBITMQ_FORCE_BOOT = yes:

    spec:
      containers:
      - env:
        - name: RABBITMQ_FORCE_BOOT # New Line 1 Added
          value: "yes"              # New Line 2 Added

And that should fix the issue also... please first try to do it in a proper way as is explained above by Ulli.

score 1 · Answer 4 · answered Dec 08 '21 at 06:44

In my case solution was simple

Step1: Downscale the statefulset it will not delete the PVC.

kubectl scale statefulsets rabbitmq-1-rabbitmq --namespace teps-rabbitmq --replicas=1

Step2: Access the RabbitMQ Pod.

kubectl exec -it rabbitmq-1-rabbitmq-0 -n Rabbit

Step3: Reset the cluster

rabbitmqctl stop_app
rabbitmqctl force_boot

Step4:Rescale the statefulset

  kubectl scale statefulsets rabbitmq-1-rabbitmq --namespace teps-rabbitmq --replicas=4

score 0 · Answer 5 · edited Jun 05 '20 at 17:05

I also got a similar kind of error as given below.

2020-06-05 03:45:37.153 [info] <0.234.0> Waiting for Mnesia tables for 30000 ms, 9 retries left 2020-06-05 03:46:07.154 [warning] <0.234.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,[rabbit_user,rabbit_user_permission,rabbit_topic_permission,rabbit_vhost,rabbit_durable_route,rabbit_durable_exchange,rabbit_runtime_parameters,rabbit_durable_queue]} 2020-06-05 03:46:07.154 [info] <0.234.0> Waiting for Mnesia tables for 30000 ms, 8 retries left

In my case, the slave node(server) of the RabbitMQ cluster was down. Once I started the slave node, master node's started without an error.

I tried a lot to solve the problem, in the end, I used the RabbitMQ operator. https://www.rabbitmq.com/kubernetes/operator/operator-overview.html — Amir Soleimani Borujerdi, Feb 27 '21 at 11:15

score -7 · Accepted Answer · answered Jul 19 '20 at 11:25

test this deploy:

kind: Service
apiVersion: v1
metadata:
  namespace: rabbitmq-namespace
  name: rabbitmq
  labels:
    app: rabbitmq
    type: LoadBalancer  
spec:
  type: NodePort
  ports:
   - name: http
     protocol: TCP
     port: 15672
     targetPort: 15672
     nodePort: 31672
   - name: amqp
     protocol: TCP
     port: 5672
     targetPort: 5672
     nodePort: 30672
   - name: stomp
     protocol: TCP
     port: 61613
     targetPort: 61613
  selector:
    app: rabbitmq
---
kind: Service 
apiVersion: v1
metadata:
  namespace: rabbitmq-namespace
  name: rabbitmq-lb
  labels:
    app: rabbitmq
spec:
  # Headless service to give the StatefulSet a DNS which is known in the cluster (hostname-#.app.namespace.svc.cluster.local, )
  # in our case - rabbitmq-#.rabbitmq.rabbitmq-namespace.svc.cluster.local  
  clusterIP: None
  ports:
   - name: http
     protocol: TCP
     port: 15672
     targetPort: 15672
   - name: amqp
     protocol: TCP
     port: 5672
     targetPort: 5672
   - name: stomp
     port: 61613
  selector:
    app: rabbitmq
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: rabbitmq-namespace
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_stomp].

  rabbitmq.conf: |
      ## Cluster formation. See http://www.rabbitmq.com/cluster-formation.html to learn more.
      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address?
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
      ## Set to "hostname" to use pod hostnames.
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
      ## environment variable.
      cluster_formation.k8s.address_type = hostname   
      ## Important - this is the suffix of the hostname, as each node gets "rabbitmq-#", we need to tell what's the suffix
      ## it will give each new node that enters the way to contact the other peer node and join the cluster (if using hostname)
      cluster_formation.k8s.hostname_suffix = .rabbitmq.rabbitmq-namespace.svc.cluster.local
      ## How often should node cleanup checks run?
      cluster_formation.node_cleanup.interval = 30
      ## Set to false if automatic removal of unknown/absent nodes
      ## is desired. This can be dangerous, see
      ##  * http://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
      ##  * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## See http://www.rabbitmq.com/ha.html#master-migration-data-locality
      queue_master_locator=min-masters
      ## See http://www.rabbitmq.com/access-control.html#loopback-users
      loopback_users.guest = false
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: rabbitmq-namespace
spec:
  serviceName: rabbitmq
  replicas: 3
  selector:
    matchLabels:
      name: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
        name: rabbitmq
        state: rabbitmq
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      containers:        
      - name: rabbitmq-k8s
        image: rabbitmq:3.8.3
        volumeMounts:
          - name: config-volume
            mountPath: /etc/rabbitmq
          - name: data
            mountPath: /var/lib/rabbitmq/mnesia
        ports:
          - name: http
            protocol: TCP
            containerPort: 15672
          - name: amqp
            protocol: TCP
            containerPort: 5672
        livenessProbe:
          exec:
            command: ["rabbitmqctl", "status"]
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 10
        resources:
            requests:
              memory: "0"
              cpu: "0"
            limits:
              memory: "2048Mi"
              cpu: "1000m"
        readinessProbe:
          exec:
            command: ["rabbitmqctl", "status"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        imagePullPolicy: Always
        env:
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: RABBITMQ_USE_LONGNAME
            value: "true"
          # See a note on cluster_formation.k8s.address_type in the config file section
          - name: RABBITMQ_NODENAME
            value: "rabbit@$(HOSTNAME).rabbitmq.$(NAMESPACE).svc.cluster.local"
          - name: K8S_SERVICE_NAME
            value: "rabbitmq"
          - name: RABBITMQ_ERLANG_COOKIE
            value: "mycookie"      
      volumes:
        - name: config-volume
          configMap:
            name: rabbitmq-config
            items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
            - key: enabled_plugins
              path: enabled_plugins
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - "ReadWriteOnce"
      storageClassName: "default"
      resources:
        requests:
          storage: 3Gi

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq 
  namespace: rabbitmq-namespace 
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: rabbitmq-namespace 
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: rabbitmq-namespace
subjects:
- kind: ServiceAccount
  name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader

To be honest, this is not really helpful. Why would somebody want to just try it out, along with a bunch of configuration which is not relevant to him/her? — Haris Osmanagić, Aug 02 '21 at 11:18
@HarisOsmanagić you can edit configmap depends on your usage. — Amir Soleimani Borujerdi, Aug 03 '21 at 05:02
No, I can't, because there's too much in it. Everybody would appreciate seeing the exact parameter which fixes an issue, and not having to experiment with another config map. — Haris Osmanagić, Aug 03 '21 at 09:36

Rabbit mq - Error while waiting for Mnesia tables

6 Answers6

Linked