How can I create many kafka topics during spring-boot application start up?

Question

I have this configuration:

@Configuration
public class KafkaTopicConfig {

    private final TopicProperties topics;

    public KafkaTopicConfig(TopicProperties topics) {
        this.topics = topics;
    }

    @Bean
    public NewTopic newTopicImportCharge() {
        TopicProperties.Topic topic = topics.getTopicNameByType(MessageType.IMPORT_CHARGES.name());
        return new NewTopic(topic.getTopicName(), topic.getNumPartitions(), topic.getReplicationFactor());
    }

    @Bean
    public NewTopic newTopicImportPayment() {
        TopicProperties.Topic topic = topics.getTopicNameByType(MessageType.IMPORT_PAYMENTS.name());
        return new NewTopic(topic.getTopicName(), topic.getNumPartitions(), topic.getReplicationFactor());
    }

    @Bean
    public NewTopic newTopicImportCatalog() {
        TopicProperties.Topic topic = topics.getTopicNameByType(MessageType.IMPORT_CATALOGS.name());
        return new NewTopic(topic.getTopicName(), topic.getNumPartitions(), topic.getReplicationFactor());
    }
}

I can add 10 differents topics to TopicProperties. And I don't want create each similar bean manually. Does some way exist for create all topic in spring-kafka or only spring?

score 11 · Accepted Answer · edited Dec 06 '21 at 02:10

Use an admin client directly; you can get a pre-built properties map from Boot's KafkaAdmin.

@SpringBootApplication
public class So55336461Application {

    public static void main(String[] args) {
        SpringApplication.run(So55336461Application.class, args);
    }

    @Bean
    public ApplicationRunner runner(KafkaAdmin kafkaAdmin) {
        return args -> {
            AdminClient admin = AdminClient.create(kafkaAdmin.getConfigurationProperties());
            List<NewTopic> topics = new ArrayList<>();
            // build list
            admin.createTopics(topics).all().get();
        };
    }
}

EDIT

To check if they already exist, or if the partitions need to be increased, the KafkaAdmin has this logic...

private void addTopicsIfNeeded(AdminClient adminClient, Collection<NewTopic> topics) {
    if (topics.size() > 0) {
        Map<String, NewTopic> topicNameToTopic = new HashMap<>();
        topics.forEach(t -> topicNameToTopic.compute(t.name(), (k, v) -> t));
        DescribeTopicsResult topicInfo = adminClient
                .describeTopics(topics.stream()
                        .map(NewTopic::name)
                        .collect(Collectors.toList()));
        List<NewTopic> topicsToAdd = new ArrayList<>();
        Map<String, NewPartitions> topicsToModify = checkPartitions(topicNameToTopic, topicInfo, topicsToAdd);
        if (topicsToAdd.size() > 0) {
            addTopics(adminClient, topicsToAdd);
        }
        if (topicsToModify.size() > 0) {
            modifyTopics(adminClient, topicsToModify);
        }
    }
}

private Map<String, NewPartitions> checkPartitions(Map<String, NewTopic> topicNameToTopic,
        DescribeTopicsResult topicInfo, List<NewTopic> topicsToAdd) {

    Map<String, NewPartitions> topicsToModify = new HashMap<>();
    topicInfo.values().forEach((n, f) -> {
        NewTopic topic = topicNameToTopic.get(n);
        try {
            TopicDescription topicDescription = f.get(this.operationTimeout, TimeUnit.SECONDS);
            if (topic.numPartitions() < topicDescription.partitions().size()) {
                if (LOGGER.isInfoEnabled()) {
                    LOGGER.info(String.format(
                        "Topic '%s' exists but has a different partition count: %d not %d", n,
                        topicDescription.partitions().size(), topic.numPartitions()));
                }
            }
            else if (topic.numPartitions() > topicDescription.partitions().size()) {
                if (LOGGER.isInfoEnabled()) {
                    LOGGER.info(String.format(
                        "Topic '%s' exists but has a different partition count: %d not %d, increasing "
                        + "if the broker supports it", n,
                        topicDescription.partitions().size(), topic.numPartitions()));
                }
                topicsToModify.put(n, NewPartitions.increaseTo(topic.numPartitions()));
            }
        }
        catch (@SuppressWarnings("unused") InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        catch (TimeoutException e) {
            throw new KafkaException("Timed out waiting to get existing topics", e);
        }
        catch (@SuppressWarnings("unused") ExecutionException e) {
            topicsToAdd.add(topic);
        }
    });
    return topicsToModify;
}

Good answer, but when I start application the second time I get an error: `Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicExistsException: Topic 'my-topic' already exists.` — ip696, Mar 28 '19 at 11:42
So, check if they exist first. I added the code that the admin uses to do that. — Gary Russell, Mar 28 '19 at 13:03

score 3 · Answer 2 · answered Oct 14 '21 at 07:37

3

Currently we can just use KafkaAdmin.NewTopics

Spring Doc

answered Oct 14 '21 at 07:37

ByeBye

6,650
5
30
63

How can I create many kafka topics during spring-boot application start up?

2 Answers2

Linked