12

I am using testcontainers.org with KafkaContainer.

Currently, I use kafka-topics to create a topic after starting the container:

kafkaContainer.execInContainer("/bin/sh", "-c", "/usr/bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topicName");

As this takes around 3-5 seconds per topic, I am wondering, if there is a more efficient way to create multiple topics. Or is there a simple switch to autocreate topics on demand?

Jochen Christ
  • 697
  • 6
  • 13

2 Answers2

14

For reference, using the AdminClient was the most efficient.

Here is an example:

  private static void createTopics(String... topics) {
    var newTopics =
        Arrays.stream(topics)
            .map(topic -> new NewTopic(topic, 1, (short) 1))
            .collect(Collectors.toList());
    try (var admin = AdminClient.create(Map.of(BOOTSTRAP_SERVERS_CONFIG, getKafkaBrokers()))) {
      admin.createTopics(newTopics);
    }
  }

  private static String getKafkaBrokers() {
    Integer mappedPort = kafkaContainer.getFirstMappedPort();
    return String.format("%s:%d", "localhost", mappedPort);
  }
Jochen Christ
  • 697
  • 6
  • 13
2

  1. Use wurstmeister/kafka container with KAFKA_CREATE_TOPICS environment variable

  2. You could use a higher level Kafka client like Spring-Kafka or Dropwizard-Kafka which offer topic creation.

  3. Otherwise, use AdminClient directly

It's recommended not to enable auto topic creation on the brokers because it then has a default partition count and replication factor

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • 4
    FYI Testcontainers uses Confluent's image for Kafka and also configured to auto-create topics. – bsideup Dec 04 '19 at 08:25
  • 1
    @bsideup As mentioned, auto-create on the broker gives default settings defined by the broker. The wurstmeister container allows for setting individiual replication and partition counts per topic. – OneCricketeer Dec 04 '19 at 16:35