0

I set up a kafka s3 connector but it fails to consume data from kafa due to error:

The coordinator is not available

The kafka is a single node cluster and seems to work fine with other consumers e.g. offset explorer can read data from the topic.

I did check similar questions asked in stackoverflow and all q/a points to offsets.topic.replication.factor should be manually set to 1 instead of default 3 in a single node cluster.

In my case, I checked the topic and it is set to 1.

./kafka-topics.sh --describe --zookeeper broker:2181 --topic 202208.topic.test
.v1
Topic: 202208.topic.test     PartitionCount: 1       ReplicationFactor: 1    Configs:
        Topic: 202208.topic.test     Partition: 0    Leader: 1       Replicas: 1     Isr: 1

The detailed message is as follows:

[2022-08-31 15:46:52,843] DEBUG [Consumer clientId=connector-consumer-s3-sink-0, groupId=connect-s3-sink] Updating last seen epoch from 0 to 0 for partition
prod.master.pxv.trade.eod.v1-0 (org.apache.kafka.clients.Metadata:178)
[2022-08-31 15:46:52,844] DEBUG [Consumer clientId=connector-consumer-s3-sink-0, groupId=connect-s3-sink] Updated cluster metadata updateVersion 106 to MetadataCache{clusterId='hWeMZOpIQ_iC5-iev3lZMQ', nodes=[broker:9092 (id: 1 rack: null)], partitions=[PartitionInfoAndEpoch{partitionInfo=Partition
(topic = 202208.topic.test, partition = 0, leader = 1, replicas = [1], isr = [1], offlineReplicas = []), epoch=0}], controller=broker:9092 (id: 1 rack: null)} (org.apache.kafka.clients.Metadata:263)
[2022-08-31 15:46:52,844] DEBUG [Consumer clientId=connector-consumer-s3-sink-0, groupId=connect-s3-sink] Sending FindCoordinator request to broker broker:9092 (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:727)
[2022-08-31 15:46:52,864] DEBUG [Consumer clientId=connector-consumer-s3-sink-0, groupId=connect-s3-sink] Received FindCoordinator response ClientResponse(receivedTimeMs=1661960812864, latencyMs=20, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=3, clientId=connector-consumer-s3-sink-0, correlationId=211), responseBody=FindCoordinatorResponseData(throttleTimeMs=0, errorCode=15, errorMessage='The coordinator is not available.', nodeId=-1, host='', port=-1)) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:741)
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Jin Ma
  • 169
  • 2
  • 12
  • Please clarify how you start Kafka and connect – OneCricketeer Sep 07 '22 at 14:48
  • Hi this is how I started the connector: connector-standalone /plugins/connector.properties /plugins/s3-sink.properties As for KAFKA I did find offsets.topic.replication.factor=5 in server.properties however, if this cannot be changed, anything I can change in the connector side to match it as we cannot assume all kafka cluster is single node cluster and offsets.topic.replication.factor must be set to 1 – Jin Ma Sep 14 '22 at 20:08
  • If you only have a single broker, then yes, you are required to edit all properties files to use 1 replica. Broker properties cannot be managed from Connect (plus, you ideally don't run Connect on the same machine as a broker in a production Kafka environment). The reason you cannot consume data is because that offsets topic cannot be created, therefore consumer group management will (silently?) fail (or you have other logs that indicate the same, separate from the Controller log you've copied) – OneCricketeer Sep 14 '22 at 21:38
  • @OneCricketeer why offsetexplorer or even a java application calling kafka API can successfully consume records from the same kafka topic without the need to worry about the # of replica? I'd imagine the connector is nothing but a typical consumer to read data and then dump to its destination component. Is the fix supposed to be in connector code in first place? – Jin Ma Sep 15 '22 at 01:07
  • I don't think offset explorer uses consumer groups. And if you're not committing offsets in your Java app, then you shouldn't have any issues, it just might log a warning since there will be no existing consumer group – OneCricketeer Sep 15 '22 at 14:44
  • Thank you for the elaboration. My follow up question is if in a production environment, the replica factor is not 1 or default 3, how kafka connect can cope with any server side configuration? – Jin Ma Sep 15 '22 at 14:56
  • @OneCricketeer Hi any chance if you could take a look at https://stackoverflow.com/questions/73722717/kafka-started-in-docker-container-cannot-be-accessed-by-offsetexployer-running-o Thanks! – Jin Ma Sep 15 '22 at 15:04
  • Kafka Connect needs the `__consumer_offsets` topic to be available. If you set `offsets.topic.replication.factor` to be at least the number of running brokers (as well as the replication factor of the 3 internal connect topics), then it should be healthy... – OneCricketeer Sep 15 '22 at 18:03

1 Answers1

0

did check similar questions asked in stackoverflow and all q/a points to offsets.topic.replication.factor should be manually set to 1 instead of default 3 in a single node cluster.

Correct

More specifically, if you are running a single broker, and any consumer tries to use a consumer group for offset management (such as Kafka Connect), the the offsets topic needs to exist.

I do not think that is related to your logs about the coordinator.

In my case, I checked the topic and it is set to 1... --topic 202208.topic.test

You're not checking the correct topic there. This is how you verify the offsets.topic.replication.factor

kafka-topics.sh --describe --bootstrap-servers broker:9092 --topic __consumer_offsets | grep 'ReplicationFactor'

Note: I changed --zookeeper flag since it is deprecated

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245