2

I'm trying to design a scalable application using the RPC pattern, involving multiple clients and servers. Based on the load, I would like to autoscale the RabbitMQ brokers (running in 2 or more docker containers).

Multiple clients and servers using a single broker is fine, but how do I configure the clients and servers to "discover" and connect to these new RabbitMQ instances as they are spun up (including the new client and server instances that are being spun up as well)?

Any server can process the request of any client as long as the connection is maintained. This is what I have in mind:

enter image description here

The components are spun up similar to the order as specified in the diagram. I want to set limits to how many clients and servers can be connected to a single rabbitMQ instance (the diagram is just an example). Any resources are welcome

RI98
  • 57
  • 8
  • The boilerplate answer is that RabbitMQ is not designed to be scaled in this manner. This is discussed in greater detail in several possible duplicate posts. – Boaz Feb 17 '22 at 14:29
  • 2
    Does this answer your question? [When to use RabbitMQ over Kafka?](https://stackoverflow.com/questions/42151544/when-to-use-rabbitmq-over-kafka) – Boaz Feb 17 '22 at 14:29
  • thank you for pointing me here @Boaz. Unfortunately, I'm required to use RabbitMQ for the final design :( – RI98 Feb 17 '22 at 14:32
  • 1
    I understand. I'm not suggesting you'd pick Kafka instead of RabbitMQ. The title of the linked post is a bit misleading. The post actually discusses the pros and cons of RabbitMQ and covers scaling too. In a nutshell, RabbitMQ is not designed to be horizontally scaled (i.e. by adding more instances) but rather to be vertically scaled (i.e. by making the one machine more powerful).You can search for a better duplicate, but the point is that the question has likely been covered in previous posts. – Boaz Feb 17 '22 at 14:35
  • Yes, I read that, it looks more and more like I need to replace RabbitMQ with a horizontally scalable queue – RI98 Feb 17 '22 at 14:38
  • 2
    It really depends on your scale. RabbitMQ is extremely efficient and can handle very high message rates without a problem. – Boaz Feb 17 '22 at 14:40
  • 1
    Thankfully, message rates right now aren't really high, so it seems like RabbitMQ will work for now. This is more for later on – RI98 Feb 17 '22 at 14:44
  • 1
    Run at least 3, and an odd number, of rabbit mq servers so they have quorum if one node is lost. – erik258 Feb 19 '22 at 00:36
  • 1
    Rabbit mq horizontally scales just fine up to a few dozen nodes. You need to make sure to configure redundancy on your queurs to be failure tolerant. Node discovery is largely automatic – erik258 Feb 19 '22 at 00:47
  • @DanielFarrell I currently have to establish a connection to these containers (running RabbitMQ) using their container-name:port. How is the discovery automatic? As in, is there some internal mechanism that RabbitMQ uses to communicate with the other nodes (RabbitMQ containers) running? if not, can you share some resources so that I can look at how I can configure it? – RI98 Feb 19 '22 at 03:21
  • Normally you just need one broker (can be a cluster consists of 3 or more nodes), inside which you can define 2 queues alternatively for 2 message delivery directions, Rabbitmq can do this. But if you just want a RPC implementation, other option like ZeroC ICE may be better choice. – Simon Nov 01 '22 at 04:34

0 Answers0