I am continuing to run Kafka Connect locally using this repo and specific folder path: https://github.com/entechlog/kafka-examples/tree/master/kafka-connect-standalone
I have added AWS MSK IAM authentication to this repo, as well as a few other small changes which allow it to run in distributed mode (in the launch shell script, I added this line exec connect-distributed /etc/"${COMPONENT}"/"${COMPONENT}".properties).
When I run the build, this is the error I get:
[2021-12-08 11:19:48,497] ERROR [Worker clientId=connect-1, groupId=connect-kafka-connect-group] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:324)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [connect-offsets]
And the kafka-connect container stops/exits. Earlier in the logs, I noticed this:
[2021-12-08 11:19:47,279] INFO Unable to use admin client to verify the cleanup policy of 'connect-offsets' topic is 'compact', either because the broker is an older version or because the Kafka principal used for Connect internal topics does not have the required permission to describe topic configurations. (org.apache.kafka.connect.util.TopicAdmin:506)
And I have made sure in my docker-compose.yml file, I have changed the kafka-connect image to image: confluentinc/cp-kafka${CONFLUENT_VERSION_TAG} and I have added ${CONFLUENT_VERSION_TAG} to all four images (and ${CONFLUENT_VERSION_TAG}=:latest).
I have also tried to create the three required internal topics manually in the launch shell script, before running the connect-distributed command which I added (I mentioned this earlier):
kafka-topics --create --bootstrap-server broker:9092 --topic connect-configs --replication-factor 1 --partitions 1 --config cleanup.policy=compact
kafka-topics --create --bootstrap-server broker:9092 --topic connect-offsets --replication-factor 1 --partitions 50 --config cleanup.policy=compact
kafka-topics --create --bootstrap-server broker:9092 --topic connect-status --replication-factor 1 --partitions 10 --config cleanup.policy=compact
and when I bash into the broker container I see:
__consumer_offsets
_schemas
along with the 3 topics I created above.
I did see this solution about the __consumer_offsets topic being corrupt: kafka connect failed to start with 'Uncaught exception in herder work thread.... position could be determined'' but I don't know how to check if that topic is corrupt or even how to create a new Kafka cluster (I have been rebuilding this image constantly anyways.
I also saw this solution: Kafka + Zookeeper: Connection to node -1 could not be established. Broker may not be available and attempted changing localhost/broker at multiple lines in the docker-compose.yml file. When I do this, I continue seeing the same error in the kafka-connect container about not being authorized for those internal topics. When I check the broker for topics, I see nothing when I try to list them.
In that same solution, I saw we can try adding this to the active listeners: KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 --link zookeeper:zookeeper confluent/kafka and so I tried that, and the broker container just exist saying it can't parse that line in the docker-compose.yml file.
So at this point I'm thinking there is an issue with the broker communicating with kafka connect, or something doesn't have the right permissions to create these three internal topics in the broker? Or, it can create them, but can't later access them? Not likely but is there a problem with the broker's version being too old? That wouldn't make sense since I used :latest.
One of the answers to my other question here Running Kafka connect in standalone mode, having issues with offsets is saying that the offset topics are for Source connectors. Well this certainly isn't a source connector and I couldn't tell you why kafka thinks my connector is a source connector (it's a sink connector). The answer also says that sink connectors use consumer groups. I have no idea what that means insofar as my problem is concerned.
All I know is that I must specify my offset topic (which I did manually as above, as well as in my docker-compose file), and that I don't know where this topic was created, if it was created at all, and where it is supposed to actually exist. Lastly, I have no idea why I am not authorized to access this topic if my kafka cluster created it locally.
I have tried everything I can imagine and I keep coming up on empty. Some assistance is greatly appreciated
