4

Assume that I have a multi-broker (running on the same host) Kafka setup with 3 brokers and 50 topics each of which is configured to have 7 partitions and replication factor of 3.

I have 50GB of memory to spend for kafka and make sure that Kafka logs will never exceed this amount of memory so I want to configure my retention policy in order to prevent this scenario.

I have setuup a delete cleanup policy: 

log.cleaner.enable=true
log.cleanup.policy=delete

and need to configure the following properties so that the data is deleted on a weekly basis and I will never run out of memory: 

log.retention.hours
log.retention.bytes
log.segment.bytes
log.retention.check.interval.ms
log.roll.hours

These topics contain data streamed by tables on a Database that have a total size of about 10GB (but inserts, updates or deletes are constantly streamed in these topics).

How should I go about configuring the aforementioned parameters so that the data is removed every 7 days and make sure that data might be deleted in a shorter window if needed so that I won't run out of memory?

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156

2 Answers2

8

Regarding the time retention it's easy, just set it to what you need.

For the size retention, this is not trivial for several reasons:

  1. the retention limits are minimum guarantees. This means if you set log.retention.bytes to 1GB, you will always have at least 1GB to data available on disk. This does not cover the maximum size on disk the partition can take, only the lower bound.

  2. the log cleaner only runs periodically (every 5 mins by default), so in the worst case scenario, you could end up with 1GB + the amount of data that can be written in 5 minutes. Depending on your environment, that can be a lot of data.

  3. Kafka writes a few more files (mostly indexes) to disk in addition of the partition's data. While these files are usually small (10MB by default), you may have to consider them.

Ignoring the indexes, one decent heuristic you can use to estimate the max disk size of a partition is:

SIZE = segment.bytes + retention.bytes

In a normal environment it's rare all partitions exceed their limits at the same time so it's usually possible to ignore the second point.

If you want to count indexes then you need to also add segment.index.bytes twice (there are 2 indexes: offset and timestamp) for each segment.

With 3 brokers and 3 replicas, each broker will host 350 partitions. It's also probably safer to include a "fudge factor" as Kafka does not like full disk! So remove 5-10% of your total disk size, especially if you don't count indexes.

With all these gotchas in mind you should be able to find the log size you need.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Does `log.retention.bytes` need to be less than `log.segment.bytes` (or the other way around) or it doesn't matter? For example, if log.segment.bytes is set to 1GB and the log.retention.bytes to 3GB, will the retention policy be effective since no log will be greater than 1GB (and therefore will never reach 3GB)? – Giorgos Myrianthous Aug 14 '18 at 06:47
  • Also, `log.retention.bytes` is a global configuration. Does this mean that it will sum up all (inactive) log segments for all the topics across all brokers and applied if it has been reached? – Giorgos Myrianthous Aug 14 '18 at 06:53
  • `log.retention.bytes` is only applied if the topic does not have a specific `retention.bytes`, otherwise they work the same way. – Mickael Maison Aug 14 '18 at 08:22
  • The log on disk is made up of segments so for best disk size control you want segment to be smaller than the retention. With `retention.bytes=3GB` and `segment.bytes=1GB`, you will have up to 4 segments on disk. 3 segments to reach the retention and the 4th one will be the active (where data is currently written to) one – Mickael Maison Aug 14 '18 at 08:25
  • 1
    It's per partition. Also as I said in the anser, it's a lower bound limit not a maximum. So with 5 partitions, you should have at least 5GB on disk – Mickael Maison Aug 14 '18 at 08:48
3

To accomplish what you've requested, I'd probably set log.retention.hours to 168, and log.retention.bytes to ~53687091200, divided by the number of topics you are planning to use. log.segment.bytes simply determines how many bytes are in a deletable log segment - the oldest log segment is what will be deleted when cleanup runs.

However, these are broker-level settings - it's generally recommended to set retention.ms on a per-topic level instead of log.retention.hours, although the default value for that is exactly what you've asked for - 7 days.

Simon Clark
  • 624
  • 3
  • 11
  • @GiorgosMyrianthous Apologies - I failed to proofread my own answer properly! I've updated my answer to be more consistent with actual facts... – Simon Clark Aug 13 '18 at 13:56
  • To make things clear, is `log.retention.bytes` a property which essentially sums up the size of all topics? I thought that `log.retention.bytes` was a per-topic configuration so the retention would be applicable 50GB*50topics was reached. – Giorgos Myrianthous Aug 13 '18 at 13:56
  • Is there any per-topic configuration for `log.segment.bytes`? For instance some topics have 3GB of data while some other 100MB so equally dividing 50GB of memory / 50 topics wouldn't be ideal here. – Giorgos Myrianthous Aug 13 '18 at 14:07
  • I initially thought that log.retention.bytes would have been per-broker, but it appears that it's per-partition. I don't think you can set a byte limit per-topic, sadly; at least, I've not yet found a way. – Simon Clark Aug 13 '18 at 14:35