What is an approach to use multiple different kafka servers in one spring boot app?

Question

I've got a spring boot app, working with a kafka (let's say kafka#1) Now i have a case, when i need to connect to kafka server of an external service (kafka#2) and tomorrow another external service's kafka should be added.

Each of kafka#1, kafka#2, kafka#3 has separate topics. I've managed tod find that topic where a simple thing is adviced - add all servers into bootstrap.servers property, separated by comma.

I'm a little worried about server-to-topic mappings - i don't think that it's right that kafka can "ask" all servers about all topics...

What is the right approach for this?

In point of the app, mho, it would be better to have multiple configs, for example: kafka1.properties kafka2.properties kafka3.properties. Then i could create kafka beans with apropriate settings (consumer container factories & factories) and define the required factories at @KafkaListener listeners. So i could avoid any unnescessary server-topic mapping problems...

Or maybe that's odd and i just need to add bootstrap.servers at a single config file kafka.properties and don't worry? Couldn't find any information about that...

Answer 1

If all your kafka servers belong to the same cluster, it is sufficient to have a single configuration for your clients. Kafka servers communicate internally with each other and share the latest metadata on all topics across the cluster even if the topic is not located on a particular server.

When defining bootstrap.server, it is therefore enough to mention only one of the servers. This is explained in more details in the description of that config:

"A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down)."

It is recommended to list more than one server in the bootstrap servers, in case one of the servers is currently not available.

Also, the intercommunication between Kafka servers is also explained in the book "Kafka -The Definitive Guide" which can be downloaded here:

"How do the clients know where to send the requests? Kafka clients use another request type called a metadata request, which includes a list of topics the client is interested in. The server response specifies which partitions exist in the topics, the replicas for each partition, and which replica is the leader. Metadata requests can be sent to any broker because all brokers have a metadata cache that contains this information."