1

A month ago, out of the blue, Redis started to fill up the server memory fast. In order to debug the problem we have run redis-cli --bigkeys and for our surprise all the used memory was freed.

We have a cluster of 6 nodes, being 3 masters and 3 slaves, each of the masters databases are around 15GB. Each of the nodes are stored in a dedicated box with 64GB each. Redis is filling the entire memory of 64GB twice a day. We have a cron running redis-cli --bigkeys twice a day to free up the used memory.

What could be the cause?

Thank you.

Henrique
  • 509
  • 7
  • 25
  • If redis-cli --bigkeys fress your memory, most likely it is expired keys that have not been evicted yet. Check your [eviction policy](https://redis.io/topics/lru-cache#eviction-policies) and maxmemory setting. The output of [INFO](https://redis.io/commands/info) may help identify the problem. Update the question with more info if possible – LeoMurillo Jan 16 '20 at 21:05
  • See https://redis.io/commands/expire#how-redis-expires-keys as to why the SCAN in redis-cli --bigkeys would force eviction – LeoMurillo Jan 16 '20 at 21:09
  • My eviction policy is noeviction and maxmemory is set to 0. 1) In order to prevent it from filling up, should I set a maxmemory and an eviction policy to delete expired keys or just maxmemory would be sufficient? 2) What would be the recommended eviction policy? After reading the eviction policy page I'm more towards on using allkeys-random. 3) Will any eviction policy first start with the expired keys before evicting non expired keys? @LeoMurillo – Henrique Jan 16 '20 at 21:32

1 Answers1

1

Sounds like you are getting OOM command not allowed errors unless you run redis-cli --bigkeys twice a day.

If that's the case, you probably have many and/or big keys with EXPIRE, being added constantly. Expired keys are removed from memory:

  • Passively: when you try to access it and the key is found to be timed out. This is how redis-cli --bigkeys is helping you, it forces a passive removal across all the keyspace.
  • Actively: every 100 ms, it tries to remove from memory expired keys at random, never investing more than 1 ms per cycle at it, until it estimates that less than 25% of expired keys remain. The logic is not that trivial, see activeExpireCycle.

So it all points that the active expire is not able to catch up in your case.

From your comment, maxmemory=0 and maxmemory-policy=noeviction. You may want to consider setting a maxmemory value, and maxmemory-policy=noeviction to volatile-ttl (remove the key with the nearest expire time).

What this does, whenever a write commands finds you are over maxmemory, it will try to free space for the new key, based on the policy. The volatile-ttl policy would evict first any expired keys remaining. See evict.c.

You may also increase the frequency of the background tasks, to purge expired keys more often, see hz in redis.conf. You may double it to 20.

By default "hz" is set to 10. Raising the value will use more CPU when Redis is idle, but at the same time will make Redis more responsive when there are many keys expiring at the same time, and timeouts may be handled with more precision.

Also, activedefrag = yes may help, see here.

There is a new active-expire-effort redis.conf setting that would allow you to invest more CPU in active-expire, but it is not available in the latest stable release (5.0.7).

Use INFO memory to get a sense of your redis server memory status. Please update the question with this output if the above doesn't help you.

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
  • Setting maxmemory value and maxmemory-policy to volatile-ttl will allow writes in case maxmemory is reached? Will it evict only expired keys or we take the chance of Redis evicting keys that are not expired when maxmemory value is reached? Doubling the hz value seems a smart idea. I will probably start with that and see if Redis can catch up expiring the keys before filling it up. – Henrique Jan 17 '20 at 17:27
  • Yes, `volatile-ttl` will allow writes in case `maxmemory` is reached, evicting volatile keys (those with an expire set) from the first to expire (expired first) to the last ones to expire. So yes, if there is no already-expired key, it will evict keys that haven't expired yet. I would start by the hz as well, good choice. But setting maxmemory is best practice, you don't want to allow redis to grab all your system memory, or even paginating (swapping-out) memory to disk if your OS has virtual memory enabled. – LeoMurillo Jan 17 '20 at 17:37
  • Thanks, setting a higher value for the hz param solved the issue. Doubling helped, but I will probably have to set it to 30 or 40 in order to keep the expiration from filling the memory. – Henrique Jan 20 '20 at 17:23