3

Lets say I have one kafka broker configured with one partition 

log.retention.bytes=80000 
log.retention.hours=6

What will happen if I try to send a record with the producer api to a broker and the log of the topic got full before the retention period?

Will my message get dropped? Or will kafka free some space from the old messages and add mine?

How can I know if a topic is getting full and logs are being deleted before being consumed?
Is there a way to monitor or expose a metric when a topic is getting full?

Mehul Gupta
  • 440
  • 5
  • 23
iah10
  • 157
  • 3
  • 12

2 Answers2

4

What will happen if I try to send a record with the producer api to a broker and the log of the topic got full before the retention period? Will my message get dropped? Or will kafka free some space from the old messages and add mine?

cleanup.policy property from topic config which by default is delete, says that "The delete policy will discard old segments when their retention time or size limit has been reached."

So, if you send record with producer api and topic got full, it will discard old segments.

How can I know if a topic is getting full and logs are being deleted before being consumed? Is there a way to monitor or expose a metric when a topic is getting full?

You can get Partition size using below script:

/bin/kafka-log-dirs.sh --describe --bootstrap-server : --topic-list

You will need to develop a script that will run above script for fetching current size of topic and send it periodically to Datadog. In Datadog, you can create widget that will trigger appropriate action(e.g. sending email alerts) once size reaches a particular threshold.

mukesh210
  • 2,792
  • 2
  • 19
  • 41
  • Thanks for your answer! but my second part of the question is about monitoring and getting and alert somehow either via the producer api or directly from kafka metrics to know that a topic is full and some messages got dropped. – iah10 Dec 12 '18 at 10:20
  • @iah10 : I edited my post. Normally you have JMX metrics exported by default about lots of things in Kafka including metrics linked to segments. – Saïd Bouras Dec 12 '18 at 17:38
  • For JMX Metrics, you can find more info here: https://stackoverflow.com/questions/41752581/kafka-jmx-metrics-in-console – mukesh210 Dec 12 '18 at 18:15
3

It's not exactly true, a topic is never full, at least by default.

I said by default because like @Mukesh said the cleanup.policy will discard old segments when their retention time or size limit is reached, but by default there is no size limit only a time limit and the property that handle that is retention.bytes (set by default to -1).

It will let only a time limit on message, note that the retention.bytes value is set by partition so to specify a limit on a topic, you have to multiply by the numbers of partitions on that topic.

EDIT : There is a tons of metrics that kafka export (in JMX) and in thoses you can found global metrics about segments (total numbers, per topic numbers, size, rate of rolling segments etc...).

Saïd Bouras
  • 286
  • 1
  • 4