3

I have a Kafka topic which has multiple consumer groups. I need the messages on the topic to not be removed when their persistence duration expires if they have not been read by all consumer groups.
Is it possible to setup additional persistence rules beyond the duration? I need the messages to always stay on a topic if they have never been consumed.
Would it be possible to "refresh" the timeout on a message if it has not been consumed and it's duration expires?

Ben Joubert
  • 31
  • 3
  • you can configure the retention policy *per topic* so for that specific one you can set it to indefinite or very long time – Paizo May 29 '18 at 08:54
  • I see that retention policies are generally based on time or size. The problem with this is that if the time on a message expires without it ever being consumed, it would be lost. Same for the size policy. Indefinite or very long duration retention is also not an option due to the amount of data I am working with. – Ben Joubert May 29 '18 at 08:59
  • This might help to undertand the concept: https://stackoverflow.com/a/28594172/2143846 – Yuval Simhon May 29 '18 at 09:31

1 Answers1

1

This is not possible in Kafka. Kafka - unlike many more traditional message brokers - does not track which message has been or has not been consumed. This is the responsibility of the consumers. And because the broker doesn't track this, it cannot do the topic cleanup based on this.

In some cases, you can use compacted topics which will keep the last message for each key. Thanks to that even a consumer which connected late might be able to recover the state. But this works only in specific data types such as state changes etc.

Jakub
  • 3,506
  • 12
  • 20
  • Okay I see now how they have the responsibilities structured. Could you perhaps point me in a direction as to how I can better handle my situation? I need to ensure that messages have been consumed before they are deleted. – Ben Joubert May 29 '18 at 11:24
  • 2
    Maybe you should start by explaining why do you use Kafka. Kafka is a great tool, but to work great it needs a slightly specific architecture of the systems using it. So I think you have three options in general: a) Start using a different messaging system; b) Redesign your consumers and producers to work with Kafka; c) Set the retention times and sizes big enough to make sure no messages will get lost. – Jakub May 29 '18 at 11:54
  • The main reason for using Kafka is that I have multiple producers, each of which gets data from various sources. These then convert data to a general format which gets put onto a topic. There are multiple consumer groups which each need to perform their own processes on this common data. I think the key here would be to build in fail safes to ensure that all my consumer groups are running, and have a long enough retention period – Ben Joubert May 29 '18 at 12:53
  • To be honest, this doesn't sound like you have anything Kafka specific. Most messaging systems allow you to have multiple producers and maybe something like durable subscriptions would more fit your consumer model. Do you run this at scale (how many nodes / partitions does your topic have)? – Jakub May 29 '18 at 14:17
  • Currently the scale is fairly small, 3 nodes with 4 partitions. But this will grow as time goes on and I need to handle larger amounts of producers/consumers. What other messaging system would you recommend that allows me to add additional producers and consumer groups with durable subscriptions? This needs to be able to scale as I am working with a big data type of environment. – Ben Joubert May 29 '18 at 15:02
  • 1
    Well ... you can try for example ActiveMQ Artemis. It can for sure handle quite a lot producers and consumers. But I do not have any exact numbers. And I anyway don't know whether you are talking about 1000s of consumers and producers, 10_000s etc. So you would need to try. – Jakub May 29 '18 at 15:10
  • If you wanna stay with Kafka, you probably just have to increase the retention so that you can be sure that no consumer will be offline for long enough to miss some messages. Normally you anyway would have some kind of SLAs for the consumers, or? Kafka is quite good at scaling the partition size ... so this is largely question of sufficient disk space and nothing else. – Jakub May 29 '18 at 15:11