58

I am developing a mobile messaging app. I was going through technology needed and found two MQTT & Apache Kafta. To me both seems doing the same thing in the same way (in terms of subscribing & publishing to a topic).

I heard that MQTT is fit for mobiles as it is very light weight ? So basically what is the difference between these two and what are the advantage of each on other?

Nilesh
  • 2,054
  • 3
  • 23
  • 43
Manish Kumar
  • 10,214
  • 25
  • 77
  • 147

2 Answers2

68

The main motive behind Kafka is scalability.

MQTT is a protocol with public specification for lightweight client / message broker communications, allowing publish/subscribe exchanges. Multiple implementations of client libraries and brokers (Mosquitto, JoramMQ...) exist and are virtually compatible. MQTT just specifies the transport, and vaguely the application part (i.e. how data is handled and possibly stored, how clients are authorized...). The spec is not clear if data consumed on a topic is only real-time or possibly persistent. The spec doesn't state anything about how the message broker implementing MQTT could/should scale.

On the other hand, Apache Kafka is a message broker based on an internal "commit log": its focus is storing massive amounts of data on disk, and allowing consumption in real-time or later (as long as data is still available on disk). It's designed to be deployable as cluster of multiple nodes, with good scalability properties. Kafka uses its own network protocol.

So you are comparing two different things here: a standard pub/sub protocol (with multiple implementations), and a specific message storing/distributing software, vaguley of the same family with its own protocol.

I'd say that if you need to store massive amount of messages, to ensure batch processing, look more at Kafka. If you have lots of clients/apps exchanging messages in real-time on many independent topics look more at the MQTT (or even AMQP) message broker implementations.

Community
  • 1
  • 1
Samrat Das
  • 1,781
  • 1
  • 21
  • 33
  • Is kafta ideal for mobile app messageing like whatsapp? – Manish Kumar May 24 '16 at 08:40
  • why i asked because i came to knew that MQTT is `light weight` and small footprint so suitable for mobile devices. So what about kafta? – Manish Kumar May 24 '16 at 09:15
  • 1
    it is similar only, not only mqtt, their are more similar protocols for those use case. me too rely on mqtt. – Samrat Das May 24 '16 at 09:26
  • 3
    It's all about capabilities, kafka has no built in msg priority, poor security, heavy protocol. It takes some work to expose kafka on mobile app, this is typically done by adding a rest layer on top. MQTT can solve those... but now comes the flip side of it, kafka scales much better than all the mqtt impl I played with... – Olivier Refalo Dec 17 '17 at 17:56
  • Re: `its focus is storing massive amounts of data on disk`: you can enable data persistence in MQTT as well. So storing data can't be the reason to use Kafka instead of mqtt. Kafka must have some other advantages?! – ttfreeman Feb 10 '21 at 19:28
  • We are using kafka as backend for microservices infrastructure. Also NVIDIA promotes it for real-time data gathering from its deepstream framework on IoT (Jetson) devices. I wonder if MQTT is really a more lightweight client and/or protocol than kafka. Are there any comparisons online? – Blafasel42 Jul 29 '21 at 11:10
  • @ttfreeman no, you can not store data with MQTT. Persistence with MQTT is used just internally, to provide supports over reboots or network failures to QOS levels > 0, just to guarantee delivery. It is not meant to store data to rebuild commit logs. On mqtt you can (functionally) store only the last message, in case the retain flag is true. You cannot access previous messages for any user intended use. – pdenti Sep 19 '21 at 16:49
30

MQTT is a standard protocol (with many implementations). Kafka (which is also a protocol) is normally used by downloading it from the Apache website or e.g. a Confluent Docker image.

It is like comparing apples and oranges, both exist for very different reasons.

Most use cases I see in IoT environments combine both MQTT and Apache Kafka. The edge devices speak MQTT protocol (for the benefits it has in edge environments. These are then forwarded to Apache Kafka to get the events into the rest of the enterprise architecture.

You can do this either via a MQTT Broker like HiveMQ + Apache Kafka or via a MQTT Proxy (so that you don't need the MQTT Broker). Both options have trade-offs, of course.

See this example of how to combine MQTT with Apache Kafka. Or go directly to the Github code: "Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data".

I also created a live demo about how to integrate Apache Kafka and MQTT.

Kai Wähner
  • 5,248
  • 4
  • 35
  • 33
  • interesting. Thanks for the insight. NVIDIA seems to promote the use of kafka directly on their jetson devices to send data over WAN to servers. Would you recommend that also? Or would it make more sense to insert an MQTT step from the jetson device to the server? – Blafasel42 Jul 29 '21 at 11:14