0

I installed and configured confluent kafka. And kafka is running with a 1GB heap size.

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" #from /bin/kafka-server-start

I created a topic “thing-data” with only one partition and using an automated job to pump some data into this topic every 5 seconds. And every message is around 2400 bytes in size.

What I see is the smallest offset of my topic is changing too frequently. That means kafka queue is able to hold very few records at a given point in time. I had a look at the topic message log files sizes in /var/log/kafka/thing-data-0/

[hduser@laptop thing-data-0]$ ll

-rw-r--r--. 1 confluent confluent 10485760 Dec 30 17:05 00000000000000148868.index
-rw-r--r--. 1 confluent confluent   119350 Dec 30 17:05 00000000000000148868.log

[hduser@laptop thing-data-0]$ ll

-rw-r--r--. 1 confluent confluent 10485760 Dec 30 17:08 00000000000000148928.index
-rw-r--r--. 1 confluent confluent    54901 Dec 30 17:08 00000000000000148928.log

[hduser@laptop thing-data-0]$ ll

-rw-r--r--. 1 confluent confluent 10485760 Dec 30 17:12 00000000000000148988.index
-rw-r--r--. 1 confluent confluent    38192 Dec 30 17:13 00000000000000148988.log

As you can see the log files rolls over very frequently. Each time old files are marked as .deleted and getting deleted after the configured time.

Below are the configuration settings related to logs from /etc/kafka/server.properties.

log.roll.hours=168
log.retention.hours=168  #i tried with log.retention.ms as well .. :-)  
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

When I restart the kafka the files looks like below. 

-rw-r--r--. 1 confluent confluent 10485760 Dec 30 17:21 00000000000000149099.index
-rw-r--r--. 1 confluent confluent        0 Dec 30 17:21 00000000000000149099.log

I suspect something with the .index file size because it is set to the maximum ( segment.index.bytes default value is 10485760). (I suspect this because kafka cluster was working fine for almost a month)

Not sure what is going wrong for this and any help will be appreciated.

Some of the reference I have made given below.

http://kafka.apache.org/documentation/

https://stackoverflow.com/questions/28586008/delete-message-after-consuming-it-in-kafka

Community
  • 1
  • 1
Tinto James
  • 121
  • 1
  • 8
  • What other custom configs do you specify? For instance, did you specify 'log.index.interval.bytes' ? Besides, the .index file should be pre-allocated so its size is correct. – amethystic Dec 30 '16 at 08:27
  • Thanks amethystic.. Looks like you are correct about index file per-allocation. https://issues.apache.org/jira/browse/KAFKA-1554 .. Let me remove that portion from the post to avoid confusion. I haven't added log.index.interval.bytes.. So it should be defaulted to 4096 – Tinto James Dec 30 '16 at 08:57
  • Is it possible that the last modified time retrieved by File.lastModified is zero which triggers the frequent log segment deleting? – amethystic Dec 30 '16 at 09:55
  • What are the timestamps of you messages you write and what version do you use? Log rolling was triggered based on wall-clock time in earlier versions of Kafka but later version use record timestamps for log rolling. Thus, if you write messages with old timestamps using latest Kafka, log rolling can be triggered very frequently. – Matthias J. Sax Jan 01 '17 at 10:04
  • Thanks Mathias for the info. I was not aware about this. I am using kafka 0.10.0.1 . From the link https://issues.apache.org/jira/browse/KAFKA-4099 looks like time based log rolling is applied from 0.10.1.0 ... Is this correct? – Tinto James Jan 03 '17 at 08:02
  • Anmethystic, Any idea how can I check segment files File.lastModified is coming as zero ? – Tinto James Jan 03 '17 at 08:09
  • Isn't that great that logs get recicled? and don't consume disk space that is unused inside file? – Bogdan Mart Jul 07 '21 at 22:26

1 Answers1

0

Did you check for log.roll.ms—This is the primary configuration. By default, it doesn't have any value. But if present it will override log.roll.hours.

xskxzr
  • 12,442
  • 12
  • 37
  • 77
Mohit Gupta
  • 33
  • 1
  • 5