1

I have been using @KafkaListeners for the sort of Kafka event consumers that must have unique ids and every event consumer must have a separate group with a very special name. For example:

@KafkaListener(id="AmazingProductEventConsumer",
               topics = "${kafka......topic}",
               clientIdPrefix = "AmazingProductEventConsumerClientId",
               groupId = "${kafka.group.id.prefix}-${environemnt.id}-${application.name}-${kafka.......topic}-#{T(java.util.UUID).randomUUID().toString()}")
public class AmazingProductEventConsumer {
   ... methods...
}

or with Batch event listener:

public class ProductBatchEventConsumer {

   @KafkaListener(id="ProductBatchEventConsumer ",
               topics = "${kafka......topic}",
               clientIdPrefix = "ProductBatchEventConsumerClientId",
               groupId = "${kafka.group.id.prefix}-${environemnt.id}-${application.name}-${kafka.......topic}-#{T(java.util.UUID).randomUUID().toString()}")
   public void batchEventConsumer(List<Record> records) {
     .... 
   }
}

The Application is pretty large, so many consumers subscribed to the same topic, but in general, about 500-600 consumers subscribed to ~180-200 topics in multiple services. I want to avoid a boilerplate code and extract common patterns that could be used for generating these parameters, except for topics and partitions.

In Spring Boot 2.2 I used a separated BeanPostProcessor for this purpose and generated needed fields before initialization and replaced KafkaListener annotation for consumers with needed instances, but in Spring Boot 2.5.0 this feature no longer available, since KafkaListenerAnnotationBeanPostProcessor uses KafkaListener ann = AnnotatedElementUtils.findMergedAnnotation(clazz, KafkaListener.class); that synthesizes new KafkaListeners based firstly declared annotation properties.

I am curious, is there any legal way on how to generate @KafkaListener's field id, clientId, groupId, and other fields that could be extracted from other listeners by a common pattern.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Aventes
  • 569
  • 1
  • 8
  • 20
  • It's not clear what you mean; please add more clarification; also `findMergedAnnotation` has been used since spring-kafka 2.2; Boot 2.2 uses spring-kafka 2.3. – Gary Russell Jun 01 '21 at 15:14
  • Don't put code in comments; it's hard to read; edit the question instead and comment that you have done so. – Gary Russell Jun 01 '21 at 19:25
  • That's not true; as I said it was changed in 2.2.0 https://github.com/spring-projects/spring-kafka/issues/812 - https://github.com/spring-projects/spring-kafka/commit/99a25221ba7f035cfa6561949abf0eaaecd80ebe#diff-3245809c3781b50f2cbdc35cfc3349008de1142d23f380e5d48f87a8d4784934 - it's still not clear exactly what you want to achieve, perhaps add your previous BPP so I can get a better understanding. – Gary Russell Jun 01 '21 at 19:29
  • I mean that previously there was a bug that we used as a feature, in Spring Kafka 2.2.5.RELEASE method findListenerAnnotations that collected class-level KafkaListeners took annotations using AnnotationUtils.findAnnotation method - this method gets latest annotation changes, that possibly changed using Reflection in Runtime. But in version 2.5.0 this method utilizes findMergedAnnotation - which gets annotation from class loader - in raw format that excludes any changes made by reflection. I'd like to clarify how such fields can be generated legally, without reflection tricks – Aventes Jun 01 '21 at 19:31
  • sorry for this, I put code non-intentionally – Aventes Jun 01 '21 at 19:32
  • I am still having a hard time understanding - you are already using property placeholders and SpEL - what do you need that you can't do that way? – Gary Russell Jun 01 '21 at 19:34
  • I want to remove these noisy SPELs and placeholders that make code hard to read and avoid boilerplate code within KafkaListener annotations, like id, clientId, groupID and generate such information based on Class name (aka event consumer) + topics information, the rest of information will be added dynamically by the system. Does it make sense? – Aventes Jun 01 '21 at 19:37
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/233199/discussion-between-aventes-and-gary-russell). – Aventes Jun 01 '21 at 19:41

1 Answers1

2

You can use much simpler SpEL.

groupId = "#{@someBean.groupId}"

and put the placeholders and UUID generation in getGroupId() on that bean.

And you can use a custom annotation meta-annotated with @KafkaListener and your props to avoid boiler plate.

Or, if you have some other variability on each listener (other than the UUID), use

#{@someBean.groupId('${small.property}'})

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • But I see that won't help with the id and clientId - which is based on the class name - let me have a think how we can support this use case... – Gary Russell Jun 01 '21 at 20:12
  • I think [this pull request](https://github.com/spring-projects/spring-kafka/pull/1818) should satisfy your needs - no reflection needed. – Gary Russell Jun 01 '21 at 21:59
  • Yay, annotation enhancer looks awesome, in which build this feature would be available? – Aventes Jun 02 '21 at 15:49
  • Current plan is 2.7.2 only, but I might consider back-porting if you can't move to 2.7 for some reason. – Gary Russell Jun 02 '21 at 15:55
  • I use Spring boot 2.5.0 version, I suppose that these changes would be available for Spring boot builds much later, correct? I am good to go with Spring Kafka 2.7.2, this is not a problem – Aventes Jun 02 '21 at 16:23
  • 1
    Yes; 2.7.2 will come with Boot 2.5.1; I don't see a release date for it yet. Once the PR is merged, you can try it with 2.7.2-SNAPSHOT from https://repo.spring.io/snapshot I could release 2.7.2 before Boot 2.5.1 if necessary. – Gary Russell Jun 02 '21 at 16:48
  • No problem, I will prepare a workaround now and migrate to your solution at once Spring Boot 2.5.1 will be available – Aventes Jun 02 '21 at 17:19
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/233248/discussion-between-aventes-and-gary-russell). – Aventes Jun 02 '21 at 19:35