12

I've read the stateful stream processing overview and if I understand correctly, one of the main reasons why the RocksDB is being used as a default implementation of the key value store is a fact, that unlike in-memory collections, it can handle data larger than the available memory, because it can flush to disk. Both types of stores can survive application restarts, because the data is backed up as a Kafka topic.

But are there other differences? For example, I've noticed that my persistent state store creates some .log files for each topic partition, but they're all empty.

In short, I'm wondering what are the performance benefits and possible risks of replacing persistent stores with in-memory ones.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Dth
  • 1,916
  • 3
  • 23
  • 34
  • 2
    Not really an answer to your question but relevant: https://stackoverflow.com/questions/40110511/why-apache-kafka-streams-uses-rocksdb-and-if-how-is-it-possible-to-change-it – Matthias J. Sax Mar 08 '18 at 22:52

2 Answers2

8

I've got a very limited understanding of the internals of Kafka Streams and the different use cases of state stores, esp. in-memory vs persistent, but what I managed to learn so far is that a persistent state store is one that is stored on disk (and hence the name persistent) for a StreamTask.

That does not give much as the names themselves in-memory vs persistent may have given the same understanding, but something that I found quite refreshing was when I learnt that Kafka Streams tries to assign partitions to the same Kafka Streams instances that had the partitions assigned before (a restart or a crash).

That said, an in-memory state store is simply recreated (replayed) every restart which takes time before a Kafka Streams application is up and running while a persistent state store is something already materialized on a disk and the only time the Kafka Streams instance has to do to re-create the state store is to load the files from disk (not from the changelog topic that takes longer).

I hope that helps and I'd be very glad to be corrected if I'm wrong (or partially correct).

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
  • 2
    Good point! I'd just add one more observation: persistent state store saves offsets in the the .checkpoint file to mark what part of the changelog topic is in the RocksDB store. Upon restart it will start reading the changelog topic from that offset, as explained here: https://www.mail-archive.com/users@kafka.apache.org/msg26808.html So in some cases it has to do some work, not just load from the disk, but it still much less than in the case of an in-memory store, if my understanding is correct. – Dth Aug 01 '19 at 18:45
2

I don't see any real reason to swap current RocksDB store. In fact RocksDB one of the fastest k,v store: Percona benchmarks (based on RocksDB)

with in-memory ones - RocksDB already acts as in-memory with some LRU algorithms involved:

RocksDB architecture

The three basic constructs of RocksDB are memtable, sstfile and logfile. The memtable is an in-memory data structure - new writes are inserted into the memtable and are optionally written to the logfile.

But there is one more noticeable reason for choosing this implementation:

RocksDB source code

If you will look at source code ratio - there are a lot of Java api exposed from C++ code. So, it's much simpler to integrate this product in existing Java - based Kafka ecosystem with comprehensive control over store, using exposed api.

uptoyou
  • 1,427
  • 19
  • 24