How can I load balance reads on a master/slave redis setup?

Question

I am learning StackExchange.Redis, and Kubernetes, so I made a simple .net core app that reads a key/value from a Redis master+2slaves deployed on kubernetes. (so, everything, Redis and my app, run inside containers)

To connect to redis I use the syntax suggested in the doc:

ConnectionMultiplexer.Connect("server1:6379,server2:6379,server3:6379");

However, if I monitor the 3 containers with redis-cli MONITOR, the requests are processed always from the master, the 2 slaves do nothing, so there is no load balancing.

I have tried also to connect to a Kubernetes load balancer service which exposes the 3 Redis containers endpoints, the result is that when I start the .net app the request is processed randomly by one of the 3 Redis nodes, but then always on the same node. I have to restart the .net app and it will query another node but subsequent queries go always on that node.

What is the proper way to read key/values in a load balanced way using StackExchange.Redis with a master/slave Redis setup?

Thank you

Answer 1

SE.Redis has a CommandFlags parameter that is optional on every command. There are some useful and relevant options here:

• DemandPrimary
• PreferPrimary
• DemandReplica
• PreferReplica

The default behaviour is PreferPrimary; write operations bump that to DemandPrimary, and there are a very few commands that actively prefer replicas (keyspace iteration, etc).

So: if you aren't specifying CommandFlags, then right now you're probably using the default: PreferPrimary. Assuming a primary exists and is reachable, then: it will use the primary. And there can only be one primary, so: it'll use one server.

A cheap option for today would be to add PreferReplica as a CommandFlags option on your high-volume read operations. This will push the work to the replicas if they can be resolved - or if no replicas can be found: the primary. Since there can be multiple replicas, it applies a basic rotation-based load-balancing scheme, and you should start to see load on multiple replicas.

If you want to spread load over all nodes including primaries and replicas... then I'll need to add new code for that. So if you want that, please log it as an issue on the github repo.

Answer 2

As per the docs it should automatically detect master/slaves. It may be that StackExchange.Redis is detecting that all your nodes are masters, thus just selecting one using its own tiebreaker rules.

I would also check the request logs on your redis-pods if there are some invalid commands that are being sent by StackExchange.Redis, maybe you don't have the right permissions for it to detect master/slaves.

Maybe you also have Sentinel enabled and StackExchange doesn't support sentinel

If you see something is not working you can open an issue on Github

Finally, you could also try twemproxy.

Answer 3

My Redis Implementation

namespace Asset.Car.Cache
{
    public class Redis : IRedis
    {
        private readonly IConnectionMultiplexer _iConnectionMultiplexer;

        public Redis(IConnectionMultiplexer connectionMultiplexer)
        {
            _iConnectionMultiplexer = connectionMultiplexer;
        }

        public async Task<T> Get<T>(string key) where T : class
        {
            var cachedResponse = await _iConnectionMultiplexer.GetDatabase().StringGetAsync(key, CommandFlags.PreferReplica);
            return cachedResponse == RedisValue.Null ? null : JsonConvert.DeserializeObject<T>(cachedResponse);
        }

        public async Task Set<T>(string key, T value, int expiryInMins = 1440) where T : class
        {
            string serVal = JsonConvert.SerializeObject(value);
            await _iConnectionMultiplexer.GetDatabase().StringSetAsync(key, serVal, System.TimeSpan.FromMinutes(expiryInMins));
        }

        public async Task Delete(string key)
        {
            await _iConnectionMultiplexer.GetDatabase().KeyDeleteAsync(key);
        }

        public async Task FlushAll()
        {
            var endpoints = _iConnectionMultiplexer.GetEndPoints();
            var server = _iConnectionMultiplexer.GetServer(endpoints[0]);
            await server.FlushDatabaseAsync(); // allowAdmin must be true
        }
    }
}

Startup.cs -

        var redisConfig = Configuration.GetSection("Redis").Get<RedisConfig>();
        services.AddSingleton<IConnectionMultiplexer>(ConnectionMultiplexer.Connect(
            new ConfigurationOptions()
            {
                EndPoints = { redisConfig.Hosts.Primary, redisConfig.Hosts.Replica },
                SyncTimeout = redisConfig.SyncTimeout,
                ConnectRetry = redisConfig.ConnectRetry,
                Password = redisConfig.Password,
                Ssl = redisConfig.Ssl,
                AllowAdmin = redisConfig.AllowAdmin
            }));

IMPORTANT DECISION FACTOR - Wrappers like IDistributedCache and StackExchangeRedis.Extensions do not include all the functions possible in the original stackexchange.Redis library, In particular I required to delete All the keys in Redis Cache, which was not exposed in these wrappers.