1

We are expiring 1000 keys per sec by average while taking snapshot we happened to see the size of the dump is comparatively lesser than the primary since because the snapshot is excluding the expired key's memory. Since expiring keys holding much memory in our platform, is there any way that we can make redis to release the memory held for expired keys periodically. (we are using 2.8.21 engine) or latest redis engine versions will resolve this problem little efficiently. Please guide me to the correct platform if stackoverflow is not appropriate for my question.

Reclaim memory guide : https://docs.redislabs.com/latest/ri/memory-optimizations/reclaim-expired-keys-memory-faster/ ( but need suggestion, whether upgrading will help a lot or doing scan will be good as mentioned in the doc)

karthik
  • 194
  • 2
  • 18

1 Answers1

1

Expired keys are removed from memory:

  • Passively: when you try to access it and the key is found to be timed out. This is how doing a full SCAN would help you, it forces a passive removal across all the keyspace.
  • Actively: every 100 ms, it tries to remove from memory expired keys at random, never investing more than 1 ms per cycle at it, until it estimates that less than 25% of expired keys remain. The logic is not that trivial, see activeExpireCycle (2.8.21 version).

Upgrading may help as there new features/configuration settings, like activedefrag.

Please see Redis filling up memory fast, running --bigkeys free it up for solutions including eviction policy and active expiring frequency.

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
  • will check SCAN method @leomurillo and update here , thanks – karthik Feb 12 '20 at 20:19
  • Thanks, SCAN has helped to free the memory, but it takes more than six hours since we have almost 250 million keys. Do we have any other options or this is what we should expect for 250 million keys. – karthik Feb 13 '20 at 17:04
  • Consider increasing the `hz` as suggested in https://stackoverflow.com/questions/59777591/redis-filling-up-memory-fast-running-bigkeys-free-it-up. It can be done live with `CONFIG SET hz 20`. Also see https://stackoverflow.com/questions/59564896/redis-scan-how-to-maintain-a-balance-between-newcomming-keys-that-might-match/59569053#59569053 and https://stackoverflow.com/questions/60055548/is-there-any-recommended-value-of-count-for-scan-hscan-command-in-redis/60056214#60056214 on how to throttle your `SCAN`. – LeoMurillo Feb 13 '20 at 18:10
  • You can also use Lua scripting so you don't send the keys returned by SCAN over the network. – LeoMurillo Feb 13 '20 at 18:12
  • running REDIS SCAN during production hours impact the system? like blocking the system – karthik Feb 15 '20 at 15:33