0

We are starting a new project, where we are evaluating the tech stack for asynchronous communication between microservices? We are considering RabbitMQ and Kafka for this.

Can anyone shed some light on the key considerations to decide one between these twos?

Thanks

Pragmatic
  • 3,093
  • 4
  • 33
  • 62

3 Answers3

1

Selection depends upon what exactly your microservices needs. Both has something different as compared to other.

RabbitMQ in a nutshell

Who are the players:

  1. Consumer
  2. Publisher
  3. Exchange
  4. Route

The flow starts from the Publisher, which send a message to exchange, Exchange is a middleware layer that knows to route the message to the queue, consumers can define which queue they are consuming from (by defining binding), RabbitMQ pushes the message to the consumer, and once consumed and acknowledgment has arrived, message is removed from the queue. Any piece in this system can be scaled out: producer, consumer, and also the RabbitMQ itself can be clustered, and highly available.

Kafka

Who are the players

  1. Consumer / Consumer groups
  2. Producer
  3. Kafka source connect
  4. Kafka sink connect
  5. Topic and topic partition
  6. Kafka stream
  7. Broker
  8. Zookeeper

Kafka is a robust system and has several members in the game. but once you understand well the flow, this becomes easy to manage and to work with. Producer send a message record to a topic, a topic is a category or feed name to which records are published, it can be partitioned, to get better performance, consumers subscribed to a topic and start to pull messages from it, when a topic is partitioned, then each partition get its own consumer instance, we called all instances of same consumer a consumer group. In Kafka messages are always remaining in the topic, also if they were consumed (limit time is defined by retention policy) Also, Kafka uses sequential disk I/O, this approach boosts the performance of Kafka and makes it a leader option in queues implementation, and a safe choice for big data use cases.

Comparing

Use Kafka if you need

  1. Time travel/durable/commit log
  2. Many consumers for the same message
  3. High throughput
  4. Stream processing
  5. Replicability
  6. High availability
  7. Message order

Use RabbitMq if you need:

  1. flexible routing
  2. Priority Queue
  3. A standard protocol message queue

For more info

Avi
  • 1,424
  • 1
  • 11
  • 32
  • Thanks Avi for the information. My question is scoped to one single use case, ie. communication among microservices. Take an example of ecommerce application. There could be order service, catalog service, payment service. Question is, which one to use and why? – Pragmatic Nov 05 '19 at 09:35
  • @Pragmatic I think kafka would be better option **Producer send a message record to a topic, a topic is a category or feed name to which records are published, it can be partitioned, to get better performance, consumers subscribed to a topic and start to pull messages from it, when a topic is partitioned, then each partition get its own consumer instance, we called all instances of same consumer a consumer group.** as you can create topic for all microservices and it will be easy to handle them based on that, as per my advice. – Avi Nov 05 '19 at 09:47
  • @LukeBakken I have provided the original link from where I have learned about it. I'm sorry if I have there is something wrong in it. It will be much helpful if you provide some of your knowledge on this topic. It will be really help me learn from experience people like you. :) – Avi Nov 05 '19 at 13:34
  • @Avi please see my clarifications [here](https://github.com/lukebakken/lukebakken.github.io/wiki/Clarifications-for-Kafka-vs-RabbitMQ) – Luke Bakken Nov 13 '19 at 23:45
  • Also, see this - https://stackoverflow.com/a/42154452/1466825 – Luke Bakken Nov 13 '19 at 23:50
  • Much more accurate comparison: https://www.cloudkarafka.com/blog/2016-12-05-apachekafka-vs-rabbitmq.html – Luke Bakken Nov 13 '19 at 23:51
0

In order to select a message broker, I think this list could be really helpful.

Supported programming languages: You probably should pick one that supports a variety of programming languages.

Supported messaging standards: Does the message broker support any standards, such as AMQP and STOMP, or is it proprietary?

Messaging order: Does the message broker preserve the ordering of messages?

Delivery guarantees: What kind of delivery guarantees does the broker make?

Persistence: Are messages persisted to disk and able to survive broker crashes?

Durability: If a consumer reconnects to the message broker, will it receive the messages that were sent while it was disconnected?

Scalability: How scalable is the message broker?

Latency: What is the end-to-end latency?

Competing consumers: Does the message broker support competing consumers?

Seyedraouf Modarresi
  • 1,213
  • 9
  • 14
0
Kafka Rabbit MQ
It's a distributed streaming platform, working on the pub-sub model. It's a message-broker, that works on pub-sub and queue-based model.
No out of the box support for retries and DLQ Supports retries and DLQ out of the box(via DLX).
Consumers can't filter messages specifically. Topic exchange and header exchange facilitate consumer-based message filtering.
Messages are retained till their validity period. Messages are gone as soon as they are read.
Does not support scheduled or delayed message routing. Supports scheduled and delayed routing of messages.
Scales horizontally Scales vertically
Pull based approach Pull based apporach
Supports event replay with consumer groups No way to event replay
Abhinav Mehta
  • 201
  • 2
  • 4