2

I am creating a service which talks to another service in order to identify kafka topics to be listened to. The kafka topics may have different key and value types. Hence, I want to dynamically create different kafka consumers for each configuration(topic, key type, value type) where the configurations are known only at runtime. However in spring kafka, I don't see ways to pass all these parameters dynamically(at least I am not aware of any). How should I go ahead with this.

Sumit
  • 631
  • 1
  • 7
  • 12
  • 1
    How about one consumer using `ByteArrayDeserializer`, then dynamically call different deserialization methods in the consumption method? – OneCricketeer Apr 11 '21 at 15:42
  • In Kafka, the consumer consumes all messages from a topic, and your consumer is part of a consumer group. There is no way you could say I wanted to consume messages based on message attribute(s). So if you think your messages/events are different types, then you should consider topic per event type. Otherwise, you can have one consumer group and multiple message processors to pick up runtime upon message arrival. Runtime polymorphism should be the answer, I would think. – Sanju Thomas Apr 11 '21 at 23:11
  • @SanjuThomasI think my question created some confusion. One topic only has one type of data but different topics will have different type of data. My problem in to spin up a new consumer with the newly identified topic and its key type value type information. – Sumit Apr 12 '21 at 04:36
  • @OneCricketeer how to configure the topics also dynamically then ? – Sumit Apr 12 '21 at 04:37
  • You'd have to start a new thread for each consumer instance. My comment was more that the config doesnt need changed for each type of data – OneCricketeer Apr 12 '21 at 04:39

1 Answers1

4

Simply create a new listener container at runtime.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#message-listener-container

If you are using Spring Boot, you can use the ConcurrentKafkaListenerContainerFactory that it automatically configures; if not, just create and initialize the containers manually.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Hi Gary, do you have a snippet? I was looking for the same implementation for Spring Boot2 – Marco Apr 06 '22 at 12:28
  • There are several examples here; see this answer for some links https://stackoverflow.com/questions/71471941/spring-kafka-multiple-topic-for-one-class-dynamically/71483489#71483489 – Gary Russell Apr 06 '22 at 12:49
  • Links go out-of-date. It would be better if you could provide an actual example here rather than a link. – Michael Jul 16 '22 at 02:09
  • 1
    With all due respect, the links in that comment are to other answers here, which have actual examples. – Gary Russell Jul 16 '22 at 02:16