Cluster forming with Mosquitto broker

Question

I am using Mosquitto broker to implement MQTT protocol. But I am unable to find how clustering can be done in case of mosquitto brokers. Also is there any limitation on number of clients those can be served with one broker.

Dominik Obermaier · Accepted Answer · 2016-12-09T09:18:43.450

19

You can't do clustering with mosquitto.

Some other MQTT brokers out there which support clustering, including HiveMQ. HiveMQ has an elastic cluster ability with auto discovery and a distributed masterless architecture and works very well on cloud providers like AWS or Azure.

You can see a full list of all brokers which support clustering here.

Disclaimer: I'm a developer of HiveMQ, so this answer may be biased.

edited Dec 09 '16 at 09:18

answered Oct 09 '14 at 15:12

Dominik Obermaier

5,610
4
34
45

how does HiveMQ behaves on Azure? By default Azure VM's have ICMP blocked, and with a regular Mosquitto instance, we experience a lot of disconnects because of that. Can you confirm that HiveMQ works all fine on latest Azure VMs ? – qgicup Jan 19 '16 at 09:51
1

According to documentation: "mosquitto clustering is achieved on backend level (redis, amqp, etc)." – Dániel Kis Apr 01 '18 at 08:41
@DánielKis: I don't see this in the official documentation and as far as I know there is no official support for that in mosquitto. Note that even if you would use a database for "clustering", you're trading the single point of failure as it would be the database. But this is more a theoretical concern as mosquitto does not support any "backend" out of the box. – Dominik Obermaier Jun 01 '18 at 23:00
You can do clustering with mosquitto https://github.com/hui6075/mosquitto-cluster ) – podarok Jul 12 '19 at 13:52

score 11 · Answer 2 · edited May 23 '17 at 12:26

Original source of this answer: Horizontal scaling for brokers. I just added the plugin support.

Two functionalities of mosquitto broker combined could be used to setup a n node cluster.

Mosquitto bridge support.
Plugin support - mosquitto-auth-plug(1)

The bridge functionality is used ONLY to synchronize messages between all mosquitto brokers while the mosquitto-auth-plugin could be used to save authorization and ACL in a single database back end.

While setting up the bridge, notice the usage of in/out bridge topics to avoid forwarding loops(2). The pattern is of format

topic pattern [[[ out | in | both ] qos-level] local-prefix remote-prefix]

Quoting mosquitto.conf man page(3), For incoming topics, the bridge will prepend the pattern with the remote prefix and subscribe to the resulting topic on the remote broker. When a matching incoming message is received, the remote prefix will be removed from the topic and then the local prefix added. And vice versa for an outgoing topic

Sample mosquitto.conf for a 3 node cluster is below. To be load tested with mqtt-malaria(4) and more importantly the effect on clients connecting with clean_session flag to false, .

                                          +-------------------+
                       +------------------> BRIDGE BROKER     <------------+
                       |                  | 192.168.1.1       |            |
                       |                  +--------^----------+            |
                       |                           |                       |
                       |                           |                       |
                       |                           |                       |
  Broker A             |             Broker B      |                       |   Broker C
                       |                           |                       |
+----------------------+--+       +----------------+--------+           +--+----------------------+
| connection A            |       | connection B            |           | connection C            |
|                         |       |                         |           |                         |
| address 192.168.1.1:1883|       | address 192.168.1.1:1883|           | address 192.168.1.1:1883|
|                         |       |                         |           |                         |
| topic # out 2 "" A/     |       | topic # out 2 "" B/     |           | topic # out 2 "" C/     |
| topic # in 2 ""  B/     |       | topic # in 2 ""  A/     |           | topic # in 2 ""  A/     |
| topic # in 2 ""  C/     |       | topic # in 2 ""  C/     |           | topic # in 2 ""  B/     |
+----------------------^--+       +----------------^--------+           +--+----------------------+
                       |                           |                       ^
                       |                           |                       |
                       |                           |                       |
                       |                   +-------+---------+             |
                       +-------------------+ HA PROXY        +-------------+
                                           +-----^--^--------+
                                                 |  |
                                                 |  |
                                                 +  +

@hardillb Should I delete this answer or a clearer source attribution is needed?. — kiranpradeep, Apr 02 '16 at 19:37
@hardillb Done that. Poor behavior on my side. Yes, Your comment link (2) in the second para was not obvious to the reader. — kiranpradeep, Apr 02 '16 at 19:44
It's important to note that this only covers message routing between brokers, clients won't be able to resume their persistent sessions on other broker instances and won't receive their queued messages. The same is true for unfinished QoS 1 and 2 message flows. — Dominik Obermaier, Jul 27 '16 at 22:57

syberkitten · Answer 3 · 2017-05-13T19:07:14.530

There are 2 (for this time of writing) major open source projects that support large scale MQTT deployments and clustering

VerneMQ and EMQ are both written in Erlang, which is very suitable for distributed messaging, having all the replication as built in toolset part of the (Erlang) language.

VerneMQ has commercial enterprise support, while EMQ is planning to in the near future, also both projects have large companies backing them as sponsors.

VerneMQ (https://vernemq.com)
EMQ (http://emqtt.io)

You may also look into HiveMQ which is the oldest kid in the block, yet cannot be used without a payed license.

HiveMQ (http://www.hivemq.com)

score 2 · Answer 4 · answered Jan 17 '15 at 23:01

2

It's not on the list mentioned by Dominik but Solace Systems has an MQTT appliance that supports clustering. Each broker supports several hundred thousand concurrent client connections.

Disclaimer: I am a Systems Engineer at Solace

answered Jan 17 '15 at 23:01

Hans Jespersen

61
2

Dwight Spencer · Answer 5 · 2015-06-11T18:22:20.250

The way I have addressed clustering for mosquitto has been to setup a haproxy frontend that reverse proxies to several listening brokers.

I have also gone one step further to utilize DNS SD which the latest versions of mosquitto support with the -S flag.

However there has been some headaches with this design, namely in application support for dnssd and libresolv on most distros do not honor the domain and search fields in resolv.conf.

Therefore if your working around a legacy environment then I would suggest looking into either rabbitmq, hivemq, or redis pubsub for more production level solution that can be shoehorned into existing footprints.

score 1 · Answer 6 · edited May 17 '17 at 03:10

1

Your may look into Bevywise IoT Platform. It is a Commercial offering. This has an option to add as many brokers as possible to scale up. This supports MQTT and MQTT-SN protocols.

Disclaimer: I work for Bevywise Networks.

edited May 17 '17 at 03:10

Nathan Tuggy

2,237
27
30
38

answered May 17 '17 at 02:29

Ranjith Kumar Diraviyam

71
6

score 0 · Answer 7 · answered Sep 21 '20 at 14:19

0

You can also check EMQX for clustering support of MQTT.

EMQX MQTT Broker

answered Sep 21 '20 at 14:19

ᐅdevrimbaris

718
9
20

Cluster forming with Mosquitto broker

7 Answers7

Linked