0

I'm new to kafka. How can I define two(or more) Kafka consumers using configuration .properties or .yml? I'm interested in using the spring.kafka.* application's properties and also I would like to specify two different properties for two consumers using this configuration files. For example, consumerA will have spring.kafka.bootstrap-servers=localhost:9090 and consumerB spring.kafka.bootstrap-servers=localhost:9091. I have seen examples online using multiple @KafkaLister with a single application.yml where all the common properties of the @KafkaLister (consumer) beans are defined in application.yml. This is ok concept but in case there are more consumers and they have completely different configurations then all of the configs needs to be put in the @KafkaListener annotation and this will make the class long and hard to read and the .yml obsolete. Instead, I would like something like this:

spring:
  kafka:
    consumer1:
      auto-offset-reset: earliest
      spring.kafka.bootstrap-servers=localhost:9091
  kafka:
    consumer2:
      spring.kafka.bootstrap-servers=localhost:9092
      auto-offset-reset: latest
  kafka:
    consumer3:
      spring.kafka.bootstrap-servers=localhost:9093
      auto-offset-reset: latest

And also how would I then connect this configuration to the appropriate beans? I could surely define the consumer beans and then use my custom configuration files to create as many different consumers as I would like but it seems to me that I would be reinventing the wheel.

Thanks

user1796624
  • 3,665
  • 7
  • 38
  • 65

1 Answers1

2

No; you can't declare consumers like that, you have to use @KafkaListener annotations on methods or create listener containers/listeners yourself.

You can override the bootstrap servers in the listener itself...

@KafkaListener(... properties="bootstrap.servers:${some.property}")
void listen(...) {

}

You can programmatically make multiple consumers from user properties, using the technique described here...

Can i add topics to my @kafkalistener at runtime

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks for the answer, as I understand it, in a case where I would like to have a project that has consumers(not prototype beans) that use different Kafka clusters with totally different topics and different properties, then I would not at all use the predefined `spring.kafka.*` configuration properties with the .yml or .properties, but have all the config in the `@KafkaListener` annotation or manually create all the beans required for these consumers programmatically. Do I understand your answer correctly? – user1796624 Oct 12 '22 at 13:45
  • Or you can define multiple consumer factories and listener container factories (beans) and reference the latter in the `containerFactory` property (which would simplify the annotation). – Gary Russell Oct 12 '22 at 13:53