1

I have the following problem. I need to set the retention time in Kafka for certain selected topics. I found a solution where I can set it with the following command: 

kafka-topics --zookeeper localhost:2181 --alter --topic topic-name --config retention.ms=-1

I checked in Kafka's Web UI and confirmed that it got changed.

If possible, I want to set the retention time in Java myself, but I can't seem to find the appropriate class/configuration to set the time. I thought I could get the information about the retention in the ProducerConfig class, but I couldn't find it there.

Is it even possible to set the retention time in Java and if possible, how can I get it done?

Thanks in advance!

goku736
  • 372
  • 2
  • 16

2 Answers2

7

This works for me :)

    private void setRetentionTime(String topicName, int retentionTime) {
        ConfigResource resource = new ConfigResource(Type.TOPIC, topicName);

        Collection<ConfigEntry> entries = new ArrayList<>();
        entries.add(new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(retentionTime)));

        Config config = new Config(entries);
        Map<ConfigResource, Config> configs = new HashMap<>();
        configs.put(resource, config);

        AdminClient client = kafkaConfig.createAdminClient();
        client.alterConfigs(configs);
    }
goku736
  • 372
  • 2
  • 16
0

I know previously from Java you can do do a login as a client. It was not possible to change topic configuration.

Recently with the new versions of Kafka they introduced this and I think it is possible to do that.

https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html#alterConfigs-java.util.Map-org.apache.kafka.clients.admin.AlterConfigsOptions-

I now the answer sounds a bit vague, but you have to be more specific if you want complete answer. I can extend the answer if you have exact version of Kafka you have, the libraries you use to connect to kafka, etc ...

I think it is not the best idea to change server side configuration from your application. This is more infrastructural thing, and should not be done runtime while you run the application.

Borislav Markov
  • 1,495
  • 11
  • 12
  • I'll look if I can solve my problem with it. We need to create topics dynamically at runtime. Certain topics must have an endless retention time. These topics are created in Java. It does not have to be done by Java., but it has to happen automatically when creating the topic. – goku736 Oct 11 '19 at 11:16
  • You might want to check possibilities to go through zookeeper if you have zookeeper. As I said I don't know your environment. Take a look at this, it covers much from your topic: https://stackoverflow.com/questions/27036923/how-to-create-a-topic-in-kafka-through-java – Borislav Markov Oct 11 '19 at 16:33