1

The read performance of Redis cluster (version > 2.8.22 in AWS) we have is being affected lately by regular scheduled snapshots/backups. I see read operations increase in latency (or Timeouts) at the time of creation of redis backups.

As per AWS docs Redis backups with version > 2.8.22 spin a child process (in replicas) when enough memory is available to create a snapshot. So, this mean redis doesn't fork the process (of creating snapshot) when enough memory isn't available.

  • So, my question is how much is the enough memory for Redis to spin up a child process to create backups?
  • Is there a way to know whether replicas in my Redis cluster is forking a child process to create backups or not?
  • My Redis replicas has about 15 - 20% of available memory while creating the backups. Is this enough to not affect the read performance?

Some steps we took to mitigate the issue:

  1. Increase number of replicas
  2. Increase reserved-memory (to 10%).

But, both steps didn't mitigate the issue.

Does increase in reserved-memory help in improving the read performance?. As per the AWS docs (https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups.html#backups-performance) reserved-memory helps to not affect Write performance.

Other workaround I'm thinking is to add a new shard to cluster. Adding a new shard would increase the available/free memory of each Replica and thus it always guarantee (theoretically) the forking of child process to create snapshots.

But, we also don't want to have too many shards to our cluster as too many shards could reduce our current read performance.

So, are there any other steps to make the snapshots/backups creation not affect Read performance?

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
barath
  • 762
  • 1
  • 8
  • 26
  • Does [this](https://stackoverflow.com/questions/52993019/redis-out-of-memory-exceptions-but-still-have-plenty-of-memory/52998436#52998436) help ? "If you are using Redis in a very write-heavy application, while saving an RDB file on disk or rewriting the AOF log Redis may use up to 2 times the memory" – rainhacker Mar 14 '19 at 17:57
  • Thanks for the link. Seems like twice the memory is recommended for write heavy clusters. Mine is not a write heavy during the backup window time. During the backup my write rate is less than 80 - 100 items to whole cluster. So, I think there should be some other reason for this. At this point I just see this to be availability of 15 - 20% of free memory which I think should be sufficient when there were not much writes into the cluster. – barath Mar 15 '19 at 00:18
  • @barath Does increasing memory twice helps in resolving the issue?? – NeiL Feb 22 '20 at 08:21
  • @NeiL I moved on from this product nows, so don't have current working solution. But at that time yes, availability of more free memory was the solution we followed. – barath Feb 24 '20 at 04:51

0 Answers0