0

There are a lot of articles across the internet about the usage of the Kafka Streams, but almost nothing about how it's done internally.

  1. Does it use any features inside Kafka outside the standard set (let's call "standard" the librdkafka implementation)?
  2. If it saves the state inside RocksDB (or any custom StateStore), how it guarantees that the state saving and the commit are in one transaction?
  3. The same question in the case when the state is saved in the compacted log (the commit and the log updated should be in one transaction).

Thank you.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Ximik
  • 2,435
  • 3
  • 27
  • 53

1 Answers1

0

I found the answers by combining information from several threads here.

  1. It uses transactions (see https://stackoverflow.com/a/54593467/414016) which are unsupported (yet) by librdkafka.
  2. It doesn't really rely on RocksDB, instead it saves the state changes into the commit log (see https://stackoverflow.com/a/50264900/414016).
  3. It does it using the transactions as mentioned above.
Ximik
  • 2,435
  • 3
  • 27
  • 53