Stack Overflow
Stack Overflow

Questions tagged [kafka-cluster]

39 questions
39
votes
3 answers

How many Kafka controllers are there in a cluster and what is the purpose of a controller?

The Kafka controller in a Kafka cluster is in charge of managing partition leaders and replication. If there are 100 brokers in a Kafka cluster, is the controller just one Kafka broker? So out of the 100 brokers, is the controller the leader? How…
김태우
  • 1,033
  • 1
  • 12
  • 29
11
votes
2 answers

Kafka: What is the minimum number of brokers required for high availability?

Suppose I want to have highly available Kafka on production on small deployment. I have to use following configs min.insync.replicas=2 // Don't want to lose messages in case of 1 broker crash default.replication.factor=3 // Will let producer…
Alexander Minyushkin
  • 123
  • 1
  • 1
  • 6
6
votes
2 answers

In Kafka HA, why minimum number of brokers required are 3 and not 2

We are trying to implement Kafka HA using kafka cluster. While doing R&D, we found that minimum number of nodes recommended for zookeeper & kafka brokers are 3. We understand that why zookeeper should have minimum 3 nodes, because for leader…
Swap
  • 159
  • 1
  • 3
  • 10
6
votes
2 answers

Understand Kafka write speed

Given that 1. Five machine Kafka cluster 2. 1 topic with 1 partition 3. persisted storage for msgs 4. Each msg of 1 KB 5. 10 producers Now, Will the max write capability of this cluster BE EQUAL TO max write capability of one java thread onto a…
Bhuvan
  • 4,028
  • 6
  • 42
  • 84
5
votes
1 answer

Kafka : Quorum-based approach to elect the new leader?

There are two common strategies for keeping replicas in sync, primary-backup replication and quorum-based replication as stated here In primary-backup replication, the leader waits until the write completes on every replica in the group before…
emilly
  • 10,060
  • 33
  • 97
  • 172
4
votes
4 answers

Error while setting up a multi-broker cluster on Local Machine

Using Kafka on Ubuntu: Zookeeper started Kafka started Topic Created Producer started Consumer started Messages are delivering fine from producer to consumer I created 2 new server.properties files as: server-1.properties…
Rahul Sharma
  • 43
  • 1
  • 5
3
votes
1 answer

Mirrored repartition topics keep increasing in size

We are using MirrorMaker to backup topics. We noticed that repartition topics created by Kafka Streams applications seem to keep increasing in size in the target cluster compared to the source cluster. This makes sense according to the documentation…
Martin Kamp Jensen
  • 157
  • 3
  • 10
3
votes
1 answer

How to make a multi-regional Kafka/Zookeeper cluster using multiple Google Kubernetes Engine (GKE) clusters?

I have 3 GKE clusters sitting in 3 different regions on Google Cloud Platform. I would like to create a Kafka cluster which has one Zookeeper and one Kafka node (broker) in every region (each GKE cluster). This set-up is intended to survive regional…
Amit Yadav
  • 4,422
  • 5
  • 34
  • 79
3
votes
1 answer

spring-kafka can't work with kafka-cluster

I've configured 3 kafka cluster and I'm trying to use with spring-kafka. but when I kill a kafka， I'm not able to send other messages to queue. Kafka version 2.0.0 spring-kafka version 2.0.1 kafka-topics.sh --describe --zookeeper=zoo1:2181 print…
Simon
  • 31
  • 4
3
votes
0 answers

When the controller becomes the second node (id 2) zookeeper and Kafka not working

we have a Kafka cluster with three nodes. We have a weird problem. When the controller becomes the second node (id 2) zookeeper and Kafka not working. zookeeper config :…
Payam Heshmati
  • 53
  • 1
  • 4
2
votes
1 answer

Kafka Cluster sometimes returns no records during seek and poll

I'm experimenting with a Kafka clusters (3 nodes) and I was intending to run some tests around redundancy and availability (stopping nodes in cluster, etc) with a simple java app using the following kafka client dependency: - …
Graeme Mitchell
  • 21
  • 2
2
votes
1 answer

Kafdrop - Cannot connect to Kafka Cluster setup using bitnami/kafka

I setup a kafka cluster using bitnami kafka and zookeeper and I wanted to view this cluster or at least one broker using kafdrop. I used docker compose to build all the components. I initially followed this tutorial and then added the kafdrop config…
NothingBox
  • 345
  • 5
  • 15
2
votes
3 answers

Can kafka partitions be spread across multiple kafka cluster nodes?

My application has a list of kafka cluster nodes specified in the spring.kafka.bootstrap-servers property and listens to topics on all these nodes. If I were to create a topic on one of these nodes, with lets say 5 partitions, will these partitions…
Ananya Antony
  • 338
  • 3
  • 15
2
votes
0 answers

Kafka cluster does not work if broker with ID=0 is down

I have a Kafka cluster with 3 brokers configured as follows: Broker-0 : broker.id=0 ; listeners=PLAINTEXT://localhost:9092 ; log.dirs=/opt/data/kafka/logs-0 Broker-1 : broker.id=1 ; listeners=PLAINTEXT://localhost:9093 ;…
rh4games
  • 962
  • 2
  • 15
  • 38
1
vote
0 answers

Local kafka cluster "__consumer_offsets" leader not available

I've setup a local kafka cluster following this blog, but I can't make it work. When I use the CLI with the following commands I can't consume the messages because they don't appear. ./kafka-console-consumer.sh --bootstrap-server localhost:9093…
Kevin Heirich
  • 109
  • 2
  • 12
1
2 3