We are using Redisson 3.17.0 and redis version 6.0.8. We have redis cluster mode setup with 3 masters and each master has about 4-5 replicas. When redis master fail over happens, redisson starts throwing exceptions that it is unable to write command into connection. Even after fail over completes (which is ~30s or so), the exceptions don't stop. Only a bounce of the instance that runs redisson resolves this error. This is affecting high availability of our service. We have pingConnectionInterval set to 5000 ms. Our read mode is only from Masters.
org.redisson.client.RedisTimeoutException: Command still hasn't been written into connection! Try to increase nettyThreads setting. Payload size in bytes: 81. Node source: NodeSource [slot=10354, addr=null, redisClient=null, redirect=null, entry=null], connection: RedisConnection@1578264320 [redisClient=[addr=rediss://-:6379], channel=[id: 0xb0f98c8c, L:/-:55678 - R:-/-:6379], currentCommand=null, usage=1], command: (EVAL), params: [local value = redis.call('hget', KEYS[1], ARGV[2]); after 2 retry attempts
Following is our redisson client config:
redisClientConfig: {
endPoint: "rediss://$HOST_IP:6379"
scanInterval: 1000
masterConnectionPoolSize: 64
masterConnectionMinimumIdleSize: 24
sslEnableEndpointIdentification: false
idleConnectionTimeout: 30000
connectTimeout: 10000
timeout: 3000
retryAttempts: 2
retryInterval: 300
pingConnectionInterval: 5000
keepAlive: true
tcpNoDelay: true
dnsMonitoringInterval: 5000
threads: 16
nettyThreads: 32
}
How can redisson recover from these exceptions without a restart of the application? We tried increasing netty threads etc, but redisson does not recover from the fail over
