Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
358
votes
5 answers

Understanding Kafka Topics and Partitions

I am starting to learn Kafka. During my readings, some questions came to my mind: When a producer is producing a message, it will specify the topic it wants to send the message to. Is that right? Does it care about partitions? When a subscriber is…
Pinidbest
  • 3,866
  • 4
  • 15
  • 17
272
votes
3 answers

What determines Kafka consumer offset?

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined…
Asif Iqbal
  • 4,562
  • 5
  • 27
  • 31
182
votes
3 answers

Is key required as part of sending messages to Kafka?

KeyedMessage keyedMessage = new KeyedMessage(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed…
gaurav
  • 2,886
  • 6
  • 24
  • 26
171
votes
3 answers

Kafka: Consumer API vs Streams API

I recently started learning Kafka and end up with these questions. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world. How Stream is different as this…
sabtharishi
  • 3,141
  • 5
  • 24
  • 27
118
votes
1 answer

Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1

I am unclear why we need both session.timeout.ms and max.poll.interval.ms and when would we use one or the other or both? It seems like both settings indicate the upper bound on the time the coordinator will wait to get the heartbeat from a consumer…
Deeps
  • 1,879
  • 4
  • 18
  • 18
114
votes
3 answers

Can multiple Kafka consumers read same message from the partition

We are planning to write a Kafka consumer(java) which reads Kafka queue to perform an action which is in the message. As the consumers run independently, will the message is processed by only one consumer at a time? Else all the consumers process…
shiv455
  • 7,384
  • 19
  • 54
  • 93
101
votes
9 answers

Kafka consumer list

I need to find out a way to ask Kafka for a list of topics. I know I can do that using the kafka-topics.sh script included in the bin\ directory. Once I have this list, I need all the consumers per topic. I could not find a script in that directory,…
Tavo
  • 3,087
  • 5
  • 29
  • 44
85
votes
4 answers

What does "Rebalancing" mean in Apache Kafka context?

I am a new user to Kafka and have been trialing it for about 2-3 weeks now. I believe at the moment I have a good understanding of how Kafka works for the most part, but after attempting to fit the API for my own Kafka consumer (this is obscure but…
Jeff Gong
  • 1,713
  • 3
  • 15
  • 19
80
votes
3 answers

Kafka Msg VS REST Calls

Nowadays in microservice world, i’m seeing alot of design in my workplace that uses kafka messaging when you can achieve similar results using rest api calls between microservices. Technically you can stop using rest api calls altogether and instead…
user1955934
  • 3,185
  • 5
  • 42
  • 68
70
votes
11 answers

How to read data using Kafka Consumer API from beginning?

Please can anyone tell me how to read messages using the Kafka Consumer API from the beginning every time when I run the consumer.
Nits
  • 1,067
  • 2
  • 9
  • 10
64
votes
2 answers

difference between groupid and consumerid in Kafka consumer

I noticed that the Consumer configuration has two IDs. One is group.id (mandatory) and second one is consumer.id (not Mandatory). What is the difference between these 2 IDs?
Gnana
  • 2,130
  • 5
  • 26
  • 57
62
votes
5 answers

Delete message after consuming it in KAFKA

I am using apache kafka to produce and consume a file 5GB in size. I want to know if there is a way where the message from the topic is automatically removed after it is consumed. Do I have any way to keep track of consumed messages? I don't want…
Shaik Mujahid Ali
  • 2,308
  • 7
  • 26
  • 40
57
votes
2 answers

Topics, partitions and keys

I am looking for some clarification on the subject. In Kafka documentations I found the following: Kafka only provides a total order over messages within a partition, not between different partitions in a topic. Per-partition ordering combined with…
Igor K.
  • 915
  • 2
  • 12
  • 22
52
votes
2 answers

How to send key, value messages with the kafka console producer

I got a use case where I need to send key value messages with Kafka Console Producer. So how to achieve this through the Kafka Console Producer command?
Keshav Lodhi
  • 2,641
  • 2
  • 17
  • 23
52
votes
2 answers

What is the difference between kafka earliest and latest offset values

producer sends messages 1, 2, 3, 4 consumer receives messages 1, 2, 3, 4 consumer crashes/disconnects producer sends messages 5, 6, 7 consumer comes back up and should receive messages starting from 5 instead of 7 For this kind of result, which…
Sat
  • 3,520
  • 9
  • 39
  • 66
1
2 3
…
99 100