Schema registry failed because _schema replication factor

Question

I want to share here a study case about the schema registry service that failed when registry service is part of Kafka machine

we have 7 Kafka brokers machines in our cluster , on each machine we have schema registry service

on one of the Kafka broker we found the following error

WARN The replication factor of the schema topic _schemas is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2022-02-10 15:09:59,958] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
java.lang.IllegalArgumentException: Unable to subscribe to the Kafka topic _schemas backing this data store. Topic may not exist.

so from log above seems that schema failed because replication factor isn't 3 as expected?

so we run the Kafka described and we get the following:

kafka-topics -zookeeper kafka01:2181 --describe --topic _schemas

Topic:_schemas  PartitionCount:1    ReplicationFactor:1 Configs:cleanup.policy=compact
    Topic: _schemas Partition: 0    Leader: 3   Replicas: 3 Isr: 3

from above we indeed see that topic- _schema is with replication factor 1

based on that details we are thinking to perform the following steps:

STEP 1

in order to increase the replication factor we need to create the following json file:

example of json template:

{"version":1,
  "partitions":[
     {"topic":"signals","partition":0,"replicas":[0,1,2]},
     {"topic":"signals","partition":1,"replicas":[0,1,2]},
     {"topic":"signals","partition":2,"replicas":[0,1,2]}
]}

but in our case and regarding to our _schema topic json should be like this:

more increase-replication-factor.json

{
 "version":1,
 "partitions":[
{"topic":"_schemas","partition":0,"replicas":[2,3,1]}
 ]
}

STEP 2

Then we should run the following:

kafka-reassign-partitions --zookeeper kafka01:2181 --reassignment-json-file increase-replication-factor.json  --execute

I will appreciate to get your opinion about STEP1,STEP2 , in order to solve the problem about schema registry service

References - How to change the number of replicas of a Kafka topic?

score 1 · Accepted Answer · answered Feb 11 '22 at 00:24

If you already have schemas in that topic, then sure, those steps will work. (Note: --zookeeper flag is replaced by --bootstrap-server in latest versions of Kafka (v3.0+) )

Otherwise, on a fresh installation, you could simply delete the topic, and edit the schema-registry.properties to set kafkastore.topic.replication.factor=3 (which is the default, by the way), then restart the registry so the topic gets recreated.

Also, you really don't need more than 2 or 3 Registry servers, at most.

Schema registry failed because _schema replication factor

1 Answers1