kafka + how it can be that partitions replica are greater than ReplicationFactor that configured

Question

we have Kafka cluster with 11 brokers on Linux machines

we have topic - my_first_car , with 300 partitions , and this topic configured with replication factor = 3

we notice from

kafka-topics.sh  --zookeeper zoo01:2181 --describe

the following example: ( please look on --> ReplicationFactor:3 )

Topic:my_first_car  PartitionCount:300  ReplicationFactor:3 Configs:retention.ms=17280000,retention.bytes=27571641765
    Topic: my_first_car Partition: 0    Leader: 1009    Replicas: 1009,1007,1008            Isr: 1007,1008,1009
    Topic: my_first_car Partition: 1    Leader: 1001    Replicas: 1001,1011,1008,1010,1009  Isr: 1001,1011,1008,1010,1009
    Topic: my_first_car Partition: 2    Leader: 1001    Replicas: 1001,1002,1011,1010,1009  Isr: 1001,1002,1011,1010,1009
    Topic: my_first_car Partition: 3    Leader: 1001    Replicas: 1001,1002,1011,1003,1010  Isr: 1001,1002,1011,1003,1010
    Topic: my_first_car Partition: 4    Leader: 1002    Replicas: 1002,1011,1001            Isr: 1002,1001,1011
    Topic: my_first_car Partition: 5    Leader: 1005    Replicas: 1005,1001,1002,1003,1004  Isr: 1003,1005,1001,1002
    Topic: my_first_car Partition: 6    Leader: 1005    Replicas: 1005,1006,1002,1003,1004  Isr: 1005,1003,1002,1006
    Topic: my_first_car Partition: 7    Leader: 1005    Replicas: 1005,1003,1004            Isr: 1003,1005,1004
    Topic: my_first_car Partition: 8    Leader: 1005    Replicas: 1006,1004,1005            Isr: 1005,1004,1006
    Topic: my_first_car Partition: 9    Leader: 1005    Replicas: 1005,1006,1007,1008,1009  Isr: 1005,1006,1007,1008,1009
    Topic: my_first_car Partition: 10   Leader: 1007    Replicas: 1007,1009,1010            Isr: 1009,1007,1010
    Topic: my_first_car Partition: 11   Leader: 1008    Replicas: 1008,1011,1001            Isr: 1001,1008,1011
    Topic: my_first_car Partition: 12   Leader: 1009    Replicas: 1009,1001,1002            Isr: 1002,1001,1009
    Topic: my_first_car Partition: 13   Leader: 1010    Replicas: 1010,1002,1003            Isr: 1003,1002,1010
    Topic: my_first_car Partition: 14   Leader: 1011    Replicas: 1011,1003,1004            Isr: 1003,1004,1011
    Topic: my_first_car Partition: 15   Leader: 1001    Replicas: 1001,1004,1005            Isr: 1005,1001
    Topic: my_first_car Partition: 16   Leader: 1002    Replicas: 1002,1005,1006            Isr: 1005,1002,1006
    Topic: my_first_car Partition: 17   Leader: 1003    Replicas: 1003,1006,1007            Isr: 1003,1006,1007
    Topic: my_first_car Partition: 18   Leader: 1004    Replicas: 1004,1007,1008            Isr: 1004,1007,1008

as we can see above on some partitions we have 5 replicas and on 2 partitions we have 4 replicas

but the configuration say clearly that - ReplicationFactor:3

we also check physically the replica on machines , and we saw that partition are really replicated according to described output

so what is the way to fix this bhavior , in order to have only 3 replica and 3 Isr ?

example of expected results

Topic:my_first_car  PartitionCount:300  ReplicationFactor:3 Configs:retention.ms=17280000,retention.bytes=27571641765
    Topic: my_first_car Partition: 0    Leader: 1009    Replicas: 1009,1007,1008            Isr: 1007,1008,1009
    Topic: my_first_car Partition: 1    Leader: 1001    Replicas: 1001,1011,1008            Isr: 1001,1011,1008
    Topic: my_first_car Partition: 2    Leader: 1001    Replicas: 1001,1002,1011            Isr: 1001,1002,1011
    Topic: my_first_car Partition: 3    Leader: 1001    Replicas: 1001,1002,1011            Isr: 1001,1002,1011
    Topic: my_first_car Partition: 4    Leader: 1002    Replicas: 1002,1011,1001            Isr: 1002,1001,1011
    Topic: my_first_car Partition: 5    Leader: 1005    Replicas: 1005,1001,1002            Isr: 1003,1005,1001
    Topic: my_first_car Partition: 6    Leader: 1005    Replicas: 1005,1006,1002            Isr: 1005,1003,1002
    Topic: my_first_car Partition: 7    Leader: 1005    Replicas: 1005,1003,1004            Isr: 1003,1005,1004
    Topic: my_first_car Partition: 8    Leader: 1005    Replicas: 1006,1004,1005            Isr: 1005,1004,1006
    Topic: my_first_car Partition: 9    Leader: 1005    Replicas: 1005,1006,1007            Isr: 1005,1006,1007
    Topic: my_first_car Partition: 10   Leader: 1007    Replicas: 1007,1009,1010            Isr: 1009,1007,1010
    Topic: my_first_car Partition: 11   Leader: 1008    Replicas: 1008,1011,1001            Isr: 1001,1008,1011
    Topic: my_first_car Partition: 12   Leader: 1009    Replicas: 1009,1001,1002            Isr: 1002,1001,1009
    Topic: my_first_car Partition: 13   Leader: 1010    Replicas: 1010,1002,1003            Isr: 1003,1002,1010
    Topic: my_first_car Partition: 14   Leader: 1011    Replicas: 1011,1003,1004            Isr: 1003,1004,1011
    Topic: my_first_car Partition: 15   Leader: 1001    Replicas: 1001,1004,1005            Isr: 1005,1001,1007
    Topic: my_first_car Partition: 16   Leader: 1002    Replicas: 1002,1005,1006            Isr: 1005,1002,1006
    Topic: my_first_car Partition: 17   Leader: 1003    Replicas: 1003,1006,1007            Isr: 1003,1006,1007
    Topic: my_first_car Partition: 18   Leader: 1004    Replicas: 1004,1007,1008            Isr: 1004,1007,1008

Question - is it good option to access the Kafka machines and to remove the additional replica from disk ?

similar posts - Cannot reassign kafka topic partition

Fixing under replicated partitions in kafka

Kafka partitions out of sync on certain nodes

https://www.signifytechnology.com/blog/2019/02/apache-kafkas-distributed-system-firefighter-the-controller-broker-by-stanislav-kozlovski

https://hevodata.com/learn/kafka-replication/

Is it possible you've ran `kafka-reassign-partitions` in the past and incorrectly wrote the JSON file? — OneCricketeer, Feb 28 '22 at 18:23
I dont think so , but could be , any idea how to fix my bad state ? — jessica, Feb 28 '22 at 20:07
not familir with command to remove replica , also searched in google — jessica, Mar 01 '22 at 15:48
what we did is the reassignment for partitions with json with 3 replica on all partition , but this not helps — jessica, Mar 01 '22 at 15:50
I'm not really sure. I've only ever increased replication factors or moved replicas, not decreased. But, using that command [seems to be the correct answer](https://stackoverflow.com/questions/64523258/reduce-topic-replication-factor-with-kafka-manager-or-kafka-cli) — OneCricketeer, Mar 01 '22 at 16:25
yes this is exactly what we did , but still we the original problem — jessica, Mar 01 '22 at 16:35
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/242521/discussion-between-jessica-and-onecricketeer). — jessica, Mar 01 '22 at 16:38
Does Zoookeeper also show more than 3 replicas under `/brokers/topics` Znode? — OneCricketeer, Mar 01 '22 at 18:03
under /brokers/topics , I see only topics names , what you mean about Znode? — jessica, Mar 01 '22 at 18:32
A ZNode is a Zookeeper path... For example - `get /brokers/topics/name` shows `{"partitions":{...}, ... ,"adding_replicas":{},"removing_replicas":{},"version":3}` — OneCricketeer, Mar 01 '22 at 18:34
under partitions , I see numbers from 0 to 299 ( as the number of partitions ) — jessica, Mar 01 '22 at 18:40
Right. And each of those numbers are assigned to an array of the replicas. Copy the output into a json-formatter to see it easier — OneCricketeer, Mar 01 '22 at 18:43
under of each number I see only the --> state and under state not see any other string — jessica, Mar 01 '22 at 18:51

kafka + how it can be that partitions replica are greater than ReplicationFactor that configured

0 Answers0