Stack Overflow
Stack Overflow

Questions tagged [apache-kafka]

Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications.

Apache Kafka

Apache Kafka is an open-source distributed streaming platform. It lets you

  • Publish and subscribe to streams of data like a messaging system
  • Store streams of data in a distributed, replicated cluster
  • Process streams of data in real-time

Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. It also grants access to the complete history of the streams unlike a database, where you only see its current state.

Links

Related Tags

31865 questions
529
votes
16 answers

When to use RabbitMQ over Kafka?

I've been asked to evaluate RabbitMQ instead of Kafka but found it hard to find a situation where a message queue is more suitable than Kafka. Does anyone know use cases where a message queue fits better in terms of throughput, durability, latency,…
Joe
  • 11,983
  • 31
  • 109
  • 183
358
votes
5 answers

Understanding Kafka Topics and Partitions

I am starting to learn Kafka. During my readings, some questions came to my mind: When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions? When a subscriber is…
Pinidbest
  • 3,866
  • 4
  • 15
  • 17
315
votes
7 answers

Using Kafka as a (CQRS) Eventstore. Good idea?

Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS, eventstore. One of the main points that Kafka supports: Event capturing/storing, all HA of course. Pub/sub architecture Ability to…
Geert-Jan
  • 18,623
  • 16
  • 75
  • 137
272
votes
3 answers

What determines Kafka consumer offset?

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined…
Asif Iqbal
  • 4,562
  • 5
  • 27
  • 31
255
votes
27 answers

Is there a way to purge the topic in Kafka?

I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error: kafka.common.InvalidMessageSizeException: invalid message size Increasing the fetch.size is not ideal here, because I don't actually want…
Peter Klipfel
  • 4,958
  • 5
  • 29
  • 44
246
votes
27 answers

Leader Not Available Kafka in Console Producer

I am trying to use Kafka. All configurations are done properly but when I try to produce message from console I keep getting the following error WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE}…
Vishesh
  • 3,599
  • 4
  • 22
  • 36
183
votes
4 answers

Data Modeling with Kafka? Topics and Partitions

One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?". I've read and watched some introductory materials. In particular, take, for example, Kafka: a…
David J.
  • 31,569
  • 22
  • 122
  • 174
182
votes
3 answers

Is key required as part of sending messages to Kafka?

KeyedMessage keyedMessage = new KeyedMessage(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed…
gaurav
  • 2,886
  • 6
  • 24
  • 26
178
votes
16 answers

How to find the kafka version in linux

How to find the kafka version in linux? whether there is a way to find the installed kafka version other than mentioning the version while downloading it?
kumar
  • 2,905
  • 5
  • 22
  • 26
177
votes
9 answers

How can I send large messages with Kafka (over 15MB)?

I send String-messages to Kafka V. 0.8 with the Java Producer API. If the message size is about 15 MB I get a MessageSizeTooLargeException. I have tried to set message.max.bytesto 40 MB, but I still get the exception. Small messages worked without…
Sonson123
  • 10,879
  • 12
  • 54
  • 72
171
votes
3 answers

Kafka: Consumer API vs Streams API

I recently started learning Kafka and end up with these questions. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world. How Stream is different as this…
sabtharishi
  • 3,141
  • 5
  • 24
  • 27
171
votes
13 answers

Is Zookeeper a must for Kafka?

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using…
Paaji
  • 2,139
  • 4
  • 14
  • 11
154
votes
3 answers

Difference between Redis and Kafka

Redis can be used as realtime pub-sub just as Kafka. I am confused which one to use when. Any use case would be a great help.
Sweta Sharma
  • 2,404
  • 4
  • 21
  • 36
138
votes
3 answers

What's the purpose of Kafka's key/value pair-based messaging?

All of the examples of Kafka | producers show the ProducerRecord's key/value pair as not only being the same type (all examples show ), but the same value. For example: producer.send(new ProducerRecord("someTopic",…
smeeb
  • 27,777
  • 57
  • 250
  • 447
129
votes
3 answers

I am evaluating Google Pub/Sub vs Kafka. What are the differences?

I have not worked on kafka much but wanted to build data pipeline in GCE. So we wanted to know Kafka vs PUB/Sub. Basically I want to know how message consistency, message availability, message reliability is maintained in both Kafka and…
Naresh
  • 1,301
  • 2
  • 9
  • 9
1
2 3
…
99 100