The offset is a pointer to the most recent message that has been consumed by a consumer. So if you read 10 messages, the offset moves 10 places. offsets.retention.minutes
allows you to move the offset back to the beginning if it isn't changed within a set period of time.
To visualise it, let's assume that we put the letters a
to g
(in that order) in a Kafka topic, all at different times. Before we start consuming the messages, the offset points to the oldest message:
OFFSET: *
MESSAGES: a b c d e f g
Now we consume 3 messages (a
, b
, c
) so the offset moves:
OFFSET: *
MESSAGES: a b c d e f g
Now let's pretend we've set log.retention.minutes=10
, and we put a
and b
into the topic 11 minutes ago, but the other messages were inserted more recently. We'd see:
OFFSET: *
MESSAGES: c d e f g
Now let's set offsets.retention.minutes=1
, and pretend it has been 90 seconds since we last consumed anything. We'd see:
OFFSET: *
MESSAGES: c d e f g
because c
is now the oldest message on the topic (and the first that will be consumed).