5

I see no mention of message keys (org.apache.kafka.clients.producer.ProducerRecord.key) except that they may be used for topic partitioning.

Am I free to put whatever data I like in the key or are there some special semantics that I should conform to?

The key seems like a convenient header to put small amounts of metadata. Is this a bad idea?

Joe
  • 46,419
  • 33
  • 155
  • 245
  • They are specially useful during log compaction. https://kafka.apache.org/documentation/#compaction – Sudhesh Rajan Jun 22 '17 at 11:08
  • Thanks. If every message key is unique, do you know if the size of the key matters (e.g. 64 bytes vs 200 bytes)? – Joe Jun 22 '17 at 11:11

1 Answers1

4

I see no mention of message keys (org.apache.kafka.clients.producer.ProducerRecord.key) except that they may be used for topic partitioning.

Use of key in Kafka: 1. Partitioning 2. guarantees ordering within a partition 3. Used during log compaction to create offset map

may be more which I haven't learnt yet.. :-)

Am I free to put whatever data I like in the key or are there some special semantics that I should conform to?

Key can be of any type Null, string or any other form with valid serialization mechanism.

do you know if the size of the key matters (e.g. 64 bytes vs 200 bytes)

Size of the key will matter as it will change your overall payload size. Therefore each message utilization on the buffer will change. I can't think of any other impact.

Sudhesh Rajan
  • 367
  • 1
  • 7
  • Thanks. Log compaction seems to be pretty important! – Joe Jun 22 '17 at 12:12
  • It's true that keys are used to determine partitioning (by default at least), but after the partition is determined, the keys no longer have anything to do with ordering within a partition. The ordering within the partition is simply sequential (for a given producer) and whichever message gets there first, in the case of multiple producers. When batching and retries are considered, it only gets messier but still nothing to do with keys themselves. – Michal Borowiecki Jun 22 '17 at 18:52
  • Null keys are not allowed in the Compacted topic. – Kamal Chandraprakash Jun 23 '17 at 09:06
  • Is there any limit on kafka message key size? – Lorenzo Belli Feb 06 '19 at 17:29