I am trying to setup vernemq cluster on Kubernetes using StatefulSet. Following is my manifest file.
apiVersion: v1
kind: ConfigMap
metadata:
name: vernemq
namespace: dev-ethernet
data:
vernemq.conf: |
accept_eula = yes
plugins.vmq_passwd = off
plugins.vmq_acl = off
allow_anonymous = on
max_client_id_size = 100
max_inflight_messages = 0
max_online_messages = 1000
message_size_limit = 0
listener.max_connections = 10000
listener.tcp.default = 127.0.0.1:1883
listener.ssl.default = 127.0.0.1:8883
listener.vmq.clustering = 0.0.0.0:44053
erlang.distribution.port_range.minimum = 9100
erlang.distribution.port_range.maximum = 9109
listener.ssl.require_certificate = on
listener.ssl.cafile = /vernemq/cert/ca.crt
listener.ssl.certfile = /vernemq/cert/server.crt
listener.ssl.keyfile = /vernemq/cert/server.key
listener.ssl.tls_version = tlsv1.2
log.console = console
log.console.level = debug
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: vernemq
namespace: dev-ethernet
labels:
app: vernemq
spec:
serviceName: vernemq
replicas: 2
podManagementPolicy: OrderedReady
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: vernemq
template:
metadata:
labels:
app: vernemq
spec:
serviceAccountName: vernemq
terminationGracePeriodSeconds: 60
containers:
- name: vernemq
image: vernemq/vernemq:1.10.4.1
imagePullPolicy: Always
#restartPolicy: OnFailure
ports:
- name: mqtt
containerPort: 1883
- name: mqtts
containerPort: 8883
- name: epmd
containerPort: 4369
- name: vmq
containerPort: 44053
- name: metrics
containerPort: 8888
- containerPort: 9100
- containerPort: 9101
- containerPort: 9102
- containerPort: 9103
- containerPort: 9104
- containerPort: 9105
- containerPort: 9106
- containerPort: 9107
- containerPort: 9108
- containerPort: 9109
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: DOCKER_VERNEMQ_DISCOVERY_KUBERNETES
value: "1"
- name: DOCKER_VERNEMQ_KUBERNETES_LABEL_SELECTOR
value: "app=vernemq"
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- /usr/sbin/vmq-admin cluster leave node=VerneMQ@${MY_POD_NAME}.vernemq.${DOCKER_VERNEMQ_KUBERNETES_NAMESPACE}.svc.cluster.local ; sleep 1 ; /usr/sbin/vmq-admin cluster leave node=VerneMQ@${MY_POD_NAME}.vernemq.${DOCKER_VERNEMQ_KUBERNETES_NAMESPACE}.svc.cluster.local -k
volumeMounts:
- name: conf
mountPath: /etc/vernemq/vernemq.conf
subPath: vernemq.conf
- name: data
mountPath: /vernemq/data
- name: cert
mountPath: /vernemq/cert
securityContext:
runAsUser: 10000
runAsGroup: 10000
fsGroup: 10000
volumes:
- name: conf
configMap:
name: vernemq
- name: cert
secret:
secretName: vernemq
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: aws-gp2
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
name: vernemq
namespace: dev-ethernet
labels:
app: vernemq
spec:
type: ClusterIP
sessionAffinity: None
ports:
- name: mqtt
port: 1883
targetPort: mqtt
- name: mqtts
port: 8883
targetPort: mqtts
selector:
app: vernemq
---
apiVersion: v1
kind: Service
metadata:
name: vernemq-headless
namespace: dev-ethernet
labels:
app: vernemq
spec:
clusterIP: None
ports:
- name: empd
port: 4369
targetPort: empd
- name: metrics
port: 8888
targetPort: metrics
selector:
app: vernemq
---
When each vernemq pod starts I see following logs
sed: cannot rename /vernemq/etc/sedmHjPDJ: Device or resource busy
/usr/sbin/start_vernemq: line 85: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 87: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 104: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 105: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 106: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 107: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 108: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 109: /vernemq/etc/vernemq.conf: Permission denied
/usr/sbin/start_vernemq: line 111: /vernemq/etc/vernemq.conf: Permission denied
config is OK
-config /vernemq/data/generated.configs/app.2020.08.07.13.35.51.config -args_file /vernemq/bin/../etc/vm.args -vm_args /vernemq/bin/../etc/vm.args
Exec: /vernemq/bin/../erts-10.7/bin/erlexec -boot /vernemq/bin/../releases/1.10.4.1/vernemq -config /vernemq/data/generated.configs/app.2020.08.07.13.35.51.config -args_file /vernemq/bin/../etc/vm.args -vm_args /vernemq/bin/../etc/vm.args -pa /vernemq/bin/../lib/erlio-patches -- console -noshell -noinput
Root: /vernemq/bin/..
13:35:53.195 [debug] Lager installed handler {lager_file_backend,"./log/error.log"} into lager_event
13:35:53.195 [debug] Lager installed handler error_logger_lager_h into error_logger
13:35:53.345 [info] alarm_handler: {set,{system_memory_high_watermark,[]}}
13:35:53.422 [info] read state from file {[{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[['VerneMQ@10.100.6.18',{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}]],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}
13:35:53.423 [info] writing state {[{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[['VerneMQ@10.100.6.18',{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}]],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}} to disk <<75,2,131,80,0,0,0,249,120,1,203,96,206,97,96,96,96,204,96,130,82,41,12,172,137,201,37,249,69,185,64,81,17,51,241,237,125,103,220,89,59,210,50,2,110,188,53,74,58,170,36,48,239,65,86,34,99,86,6,103,10,3,75,74,102,114,73,34,99,162,0,16,114,36,6,36,26,100,8,100,161,129,12,70,132,24,216,120,16,193,148,194,32,28,150,90,148,151,234,27,232,96,104,160,103,104,96,160,103,166,103,104,65,154,3,96,22,33,28,194,64,172,67,96,90,1,27,43,76,93>>
13:35:53.438 [info] Datadir ./data/meta/meta/0 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,38549145}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.445 [info] Datadir ./data/meta/meta/1 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,35929523}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.452 [info] Datadir ./data/meta/meta/2 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,31795161}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.459 [info] Datadir ./data/meta/meta/3 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,52387290}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.467 [info] Datadir ./data/meta/meta/4 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,35280518}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.475 [info] Datadir ./data/meta/meta/5 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,60988348}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.483 [info] Datadir ./data/meta/meta/6 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,43175699}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.502 [info] Datadir ./data/meta/meta/7 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,35409215}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.509 [info] Datadir ./data/meta/meta/8 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,51476514}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.516 [info] Datadir ./data/meta/meta/9 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,62879451}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.523 [info] Datadir ./data/meta/meta/10 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,54094689}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.530 [info] Datadir ./data/meta/meta/11 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,54172211}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.550 [debug] 0ms mailbox traversal, schedule next lazy broadcast in 10000ms, the min interval is 10000ms
13:35:53.577 [debug] loading modules: [vmq_plumtree,vmq_plumtree_app,vmq_plumtree_sup]
13:35:53.603 [info] Try to start vmq_plumtree: ok
13:35:53.637 [debug] datadir ./data/msgstore/1 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,47750847}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.645 [info] Opening LevelDB database at "./data/msgstore/1"
13:35:53.646 [debug] datadir ./data/msgstore/2 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,50090150}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.653 [info] Opening LevelDB database at "./data/msgstore/2"
13:35:53.653 [debug] datadir ./data/msgstore/3 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,34848743}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.660 [info] Opening LevelDB database at "./data/msgstore/3"
13:35:53.661 [debug] datadir ./data/msgstore/4 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,41593831}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.667 [info] Opening LevelDB database at "./data/msgstore/4"
13:35:53.668 [debug] datadir ./data/msgstore/5 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,45767049}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.674 [info] Opening LevelDB database at "./data/msgstore/5"
13:35:53.675 [debug] datadir ./data/msgstore/6 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,56785000}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.681 [info] Opening LevelDB database at "./data/msgstore/6"
13:35:53.682 [debug] datadir ./data/msgstore/7 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,60588967}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.683 [debug] Lager installed handler lager_backend_throttle into lager_event
13:35:53.688 [info] Opening LevelDB database at "./data/msgstore/7"
13:35:53.689 [debug] datadir ./data/msgstore/8 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,41872335}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.695 [info] Opening LevelDB database at "./data/msgstore/8"
13:35:53.696 [debug] datadir ./data/msgstore/9 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,35898829}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.703 [info] Opening LevelDB database at "./data/msgstore/9"
13:35:53.703 [debug] datadir ./data/msgstore/10 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,47214115}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.710 [info] Opening LevelDB database at "./data/msgstore/10"
13:35:53.711 [debug] datadir ./data/msgstore/11 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,36069724}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.719 [info] Opening LevelDB database at "./data/msgstore/11"
13:35:53.720 [debug] datadir ./data/msgstore/12 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,56905298}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}]
13:35:53.730 [info] Opening LevelDB database at "./data/msgstore/12"
13:35:53.730 [debug] loading modules: [vmq_generic_msg_store,vmq_generic_msg_store_app,vmq_generic_msg_store_sup,vmq_generic_msg_store_utils,vmq_storage_engine_dets,vmq_storage_engine_ets,vmq_storage_engine_leveldb]
13:35:53.767 [info] Try to start vmq_generic_msg_store: ok
13:35:53.770 [info] rename VerneMQ node from 'VerneMQ@10.100.6.18' to 'VerneMQ@10.100.6.34'
13:35:53.773 [info] writing state {[{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1},{<<105,153,35,109,220,173,228,160,165,82,37,54,41,250,215,10,226,156,138,159>>,1}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[['VerneMQ@10.100.6.34',{<<105,153,35,109,220,173,228,160,165,82,37,54,41,250,215,10,226,156,138,159>>,1}]],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}} to disk <<75,2,131,80,0,0,1,6,120,1,203,96,206,97,96,96,96,202,96,2,81,140,25,76,41,12,172,137,201,37,249,69,185,64,174,136,153,248,246,190,51,238,172,29,105,25,1,55,222,26,37,29,85,18,152,247,32,43,17,168,10,44,155,57,83,57,247,206,218,39,11,150,6,169,154,105,254,186,206,245,104,78,215,252,68,198,172,12,206,20,6,150,148,204,228,146,68,198,68,1,32,228,72,12,72,52,200,16,200,66,3,25,140,16,49,176,189,96,55,164,48,8,135,165,22,229,165,250,6,58,24,26,232,25,26,24,232,153,233,25,155,224,183,12,217,80,132,197,12,196,88,140,172,21,0,76,92,86,0>>
13:35:53.868 [info] loaded 0 subscriptions into vmq_reg_trie
13:35:53.872 [info] cluster event handler 'vmq_cluster' registered
.
.
.
15:10:21.166 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:21.166 [warning] terminate due to remote_node_not_available
15:10:21.166 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
15:10:22.168 [info] successfully connected to cluster node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'
15:10:22.171 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:22.171 [warning] terminate due to remote_node_not_available
15:10:22.171 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
15:10:23.174 [info] successfully connected to cluster node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'
15:10:23.175 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:23.175 [warning] terminate due to remote_node_not_available
15:10:23.175 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
15:10:24.178 [info] successfully connected to cluster node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'
15:10:24.181 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:24.181 [warning] terminate due to remote_node_not_available
15:10:24.181 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
15:10:24.855 [debug] started plumtree_metadata_manager exchange with 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' (<0.7531.0>)
15:10:24.864 [debug] completed metadata exchange with 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'. nothing repaired
15:10:24.874 [debug] 0ms mailbox traversal, schedule next lazy broadcast in 10000ms, the min interval is 10000ms
15:10:25.183 [info] successfully connected to cluster node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'
15:10:25.184 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:25.184 [warning] terminate due to remote_node_not_available
15:10:25.184 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
15:10:26.187 [info] successfully connected to cluster node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'
15:10:26.187 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:26.187 [warning] terminate due to remote_node_not_available
15:10:26.187 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
15:10:27.190 [info] successfully connected to cluster node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local'
15:10:27.190 [debug] connect request from unknown cluster node 'VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local'
15:10:27.190 [warning] terminate due to remote_node_not_available
15:10:27.190 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
cluster status looks good
vernemq@vernemq-0:~$ vmq-admin cluster show
+----------------------------------------------------------+---------+
| Node | Running |
+----------------------------------------------------------+---------+
| VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local | true |
+----------------------------------------------------------+---------+
| VerneMQ@vernemq-1.vernemq.dev-ethernet.svc.cluster.local | true |
+----------------------------------------------------------+---------+
But I am concerned about
15:10:27.190 [warning] terminate due to remote_node_not_available
15:10:27.190 [warning] connection to node 'VerneMQ@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms
Whats is causing nodes in cluster to disconnect ? or is it normal behavior ?
