4

My objective is it deploy Kafka connect connectors ( s3 sink) using containers, so it's important to know the details about how workers may or may not communicate directly with each others.

From what I read, I cannot be sure if workers will have a direct communication between each others ( like one worker talking to another one using REST, or whatever).

i thought at the beginning that when I'm updating a configuration using REST, the config changes are propagated using the connect_config internal topic and/or by using Kafka consumer coordinator..

But from what I read :

rest.advertised.host.name
If this is set, this is the hostname that will be given out to other workers to connect to.

Any details on how and why ( if they do) workers communicate each others ?

Yannick
  • 1,240
  • 2
  • 13
  • 25

1 Answers1

6

Yes when running in distributed mode, workers have to be able to connect to each other's REST API.

Connect allows you to use the REST API of any workers to create/update/delete a connector, however under the covers, it forwards the request to the "leader" worker.

This is described in the REST API section in the docs:

The REST API is used not only by users to monitor / manage Kafka Connect. It is also used for the Kafka Connect cross-cluster communication. Requests received on the follower nodes REST API will be forwarded to the leader node REST API. In case the URI under which is given host reachable is different from the URI which it listens on, the configuration options rest.advertised.host.name, rest.advertised.port and rest.advertised.listener can be used to change the URI which will be used by the follower nodes to connect with the leader

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68