Stack Overflow
Stack Overflow

Questions tagged [kafka-partition]

For questions specifically related to Topic Partitioning in Kafka. The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. When using this tag also include the [kafka-topic] tag and the more generic [apache-kafka] tag where possible.

The data within kafka topics are divided into multiple partitions. Partitioning helps in consuming faster and more parallelization can be achieved by splitting up the partitions across multiple brokers and multiple machines/servers. This allows multiple consumers to consume from multiple partitions and this improves the performance of data consuming application.

52 questions
9
votes
3 answers

Kafka log.segment.bytes vs log.retention.hours

I was following the book "Kafka: The Definitive Guide" First Edition to understand when log segments are deleted by the broker. As per the text I understood, a segment will not become eligible for deletion until it is closed. A segment can be closed…
Mayank Madhav
  • 429
  • 1
  • 7
  • 19
9
votes
4 answers

What is a partition leader in Apache Kafka?

Are kafka leaders partitions themselves or are they brokers? My initial understanding was that they were partitions which acted as read/write agents which then deffered their value to ISRs. However recently I have been hearing them mentioned as…
Matt
  • 113
  • 1
  • 1
  • 5
6
votes
1 answer

How Kafka guarantee the messages order while we increase the partitions in runtime?

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic…
injoy
  • 3,993
  • 10
  • 40
  • 69
4
votes
1 answer

Partition selection in Kafka

I am curious about that If I have topic A and B which they have same number of partitions, if I send message with key x to topic A it goes partition 0 let's say. When I use exactly the same key for topic B but they are independent, in topic B for…
Alpcan Yıldız
  • 671
  • 3
  • 13
  • 33
4
votes
2 answers

Is it possible to consume kafka messages using key and partition?

I am using kafka_2.12 version 2.3.0 where I am publishing data into kafka topic using partition and key. I need to find a way using which I can consume a particular message from topic using key and partition combination. That way I won't have to…
Abhishek Gharai
  • 227
  • 3
  • 15
3
votes
1 answer

Does kafka support millions of partitions?

Will we have any problem if we have millions of partitions for one topic? Due to our business requirement, we are thinking if we can make a partition for every user in kafka. We have millions of users. Any insight would be appreciated!
Moon
  • 361
  • 2
  • 10
3
votes
1 answer

Consuming kafka batch for multiple partitions

I understand that Kafka can pull events in batches. I am trying to understand this scenario: I have 4 partitions for a topic I have 1 consumer which Kafka assigns all 4 partitions to it. let's assume every batch Kafka client pulls from Kafka is 5…
toto
  • 1,197
  • 2
  • 15
  • 26
3
votes
1 answer

How to distribute messages between Kafka topics with different configuration?

I'm looking for a way how to distribute messages between two Kafka topics. In original topic I have 20 partitions with 1000000 messages per partition. I want to have a new topic with 1000 partitions and spread messages across new wider partition…
Omar
  • 47
  • 6
3
votes
2 answers

Splitting Kafka into separate topic or single topic/multiple partitions

As usual, it's bit confusing to see benefits of splitting methods over others. I can't see the difference/Pros-Cons between having Topic1 -> P0 and Topic 2 -> P0 over Topic 1 -> P0, P1 and a consumer pull from 2 topics or single topic/2…
Ahmed Alaa El-Din
  • 1,813
  • 1
  • 16
  • 19
2
votes
2 answers

Kafka consumer not able to read from all available partitions

We are noticing a weird issue with one of our prod topics(6 partitions) where our consumer (dotnet core, only 1 instance) is only able to read from 3 partitions (0,1,3). This is obviously impacting application behavior as the consumer is missing the…
AJ_NY
  • 59
  • 1
  • 8
2
votes
1 answer

What if a Kafka's consumer handles a message too long? Will Kafka reappoint this partition to another consumer and the message will doubly handled?

Suppose Kafka, 1 partition, 2 consumers.(2nd consumer is idle) Suppose the 1st one consumed a message, goes to handle it with 3 other services and suddenly sticks on one of them and miss the Kafka's timeout. Will Kafka reappoint the partition to the…
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55
2
votes
1 answer

Size of Kafka Partitions and writing to a Topic with no free disk space

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: Are all the partitions on a topic of the same size? What will happen if a Kafka message is sent on a topic and the topic is…
Saurabh Prakash
  • 2,715
  • 1
  • 11
  • 17
2
votes
2 answers

Create concurrent Kafka consumers in Camel-Kafka

I'm using camel-kafka version 2.14.3 . Below is the kafka URI : note that I…
vineeth kumar
  • 175
  • 1
  • 12
2
votes
1 answer

How to scale kafka consumer applications on the same topic with more than one worker (the same number of partitions)

I would like to understand how to have scale an application (more than a few instances) consuming messages from Kafka on the same topic. My doubt is if I have an A topic with 4 partitions and I create an @KafkaListener with the attribute concurrency…
Renato Garcia
  • 961
  • 1
  • 9
  • 13
1
vote
2 answers

Kafka flink producer does not seem to do partition selection

Kafka is sending messages to only one partition on a topic. I have used KeyedSerializationSchema in the FlinkKafkaProducer09 and passing an attribute from the event stream which is to be used to perform the hash partition selection (As I want all…
user4923462
  • 97
  • 3
  • 16
1
2 3 4