21

I try to get message by key from kafka. I found the only one solution is to use StateStore but I think it might be not a good idea. How to get message by key from kafka topic? Is it a good idea to use StateStore for this operation?

Israel Varea
  • 2,600
  • 2
  • 17
  • 24
Іван Гладуш
  • 375
  • 1
  • 2
  • 8

2 Answers2

20

Every record written to Kafka can optionally have a key (but it doesn't have to!), the key can be accessed a number of ways:

Console consumer:

$ kafka-console-consumer --bootstrap-server <servername>:9092 --topic topicname --from-beginning --property print.key=true --property key.separator=:

kafkacat:

$ kafkacat -b <servername>:9092 -C -t topicname -o beginning -K :

Java Consumer API:

ConsumerRecord#key()

Kafka isn't indexed by key though, it's indexed by offset, and optionally by timestamp. If you need to lookup a key then you'll need to materialize the data to a system that's been designed to lookup by key: relational database, key value store, or some index. You can do this pretty easily with Kafka Connect, or if you'd like to build it into your service you could use the interactive queries feature of Kafka Streams.

Chris Matta
  • 3,263
  • 3
  • 35
  • 48
  • 1
    Thank you for your response. But what do you think about StateStore, may it resolve my problem? – Іван Гладуш Sep 05 '18 at 13:36
  • 1
    The relational databases, key value stores, indexes, or interactive queries are all "state stores", essentially materializations of the records in the Kafka topic. This is the only way to index based on key, since Kafka doesn't provide that functionality, you'll have to use some other store that indexes by key. If you think about it, this is actually a feature of Kafka: the agility to use the best tool for the job. – Chris Matta Sep 05 '18 at 13:40
  • 1
    Is a good idea use a default RocksDBStore or may exist better solution? – Іван Гладуш Sep 05 '18 at 13:45
  • The interactive queries feature I linked above will actually instantiate a RocksDB state store automatically, it's a fairly simple solution, and well proven. – Chris Matta Sep 05 '18 at 13:52
  • @ChrisMatta is there a way to achieve what OP asked using KStream, considering Kafka setup has 1 partition and the key is a sequential Long type value? – Govinda Sakhare Apr 24 '20 at 14:15
5

You can't "get messages by key from Kafka".

One solution, if practical, would be to have as many partitions as keys and always route messages for a key to the same partition.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179