12

I'm trying to find out why my kafka-streams application runs out of memory. I already found out that rocksDB is consuming lots of native memory and I tried to restrict it with the following configuration:

# put index and filter blocks in blockCache to avoid letting them grow unbounded (https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-and-filter-blocks)
cache_index_and_filter_blocks = true;

# avoid evicting L0 cache of filter and index blocks to reduce performance impact of putting them in the blockCache (https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-and-filter-blocks)
pinL0FilterAndIndexBlocksInCache=true

# blockCacheSize should be 1/3 of total memory available (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#block-cache-size)
blockCacheSize=1350 * 1024 * 1024

# use larger blockSize to reduce index block size (https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#difference-of-spinning-disk)
blockSize=256 * 1024

but still the memory usage seems to grow unbounded and my container eventually gets OOMKilled.

I used jemalloc to profile the memory usage (like described here) and the result clearly shows that rocksDB is responsible but I have no clue how to further restrict the memory usage of rocksDB.

jemalloc profiling

I don't know if it is helpful, but for completeness here are statistics gathered from a running rocksdb instance:

rocksDB statistics

I'm glad for any hints

D-rk
  • 5,513
  • 1
  • 37
  • 55

2 Answers2

8

I found out what was causing this.

I thought that my kafka streams application would have only one rockDB instance. But there is one instance per stream partition. So this configuration:

blockCacheSize=1350 * 1024 * 1024

Does not necessarily mean that the rocksDB memory is restricted to 1350MB. If the application has e.g. 8 stream partitions assigned it also has 8 blockCaches and thus can take up to 1350 * 8 = ~11GB of memory.

D-rk
  • 5,513
  • 1
  • 37
  • 55
  • I thought i read somewhere that increasing block cache size reduces the number of indices and so reduces memory footprint? May I ask if you know how I can tune a rocksDB which is only writing? I need good write performance but I am not reading at all. – user997112 Sep 29 '21 at 15:13
3

Are you seeing the memory usage grow quickly or over a longer period of time?

We have found and fixed a few RocksDB resource leaks that would cause memory leaks:

There are some indications that there may be others (https://issues.apache.org/jira/browse/KAFKA-8367), either in our usage of RocksDB or in RocksDB itself.

Oh, one other idea is that if you're using iterators from the state stores, either in your processors or in Interactive Query, you have to close them.

Beyond looking for leaks, I'm afraid I don't have too much insight into diagnosing RocksDB's memory usage. You could also restrict the Memtable size, but I don't think we set it very large by default anyway.

Hope this helps,

-John

John
  • 726
  • 4
  • 5