Connecting to Redis from Azure throwing Intermittent Exceptions

Question

I have a Windows Azure Server running my C# application. It is distributed on 4 medium instances and I am using Redis for my L2 caching. The application is handling a pretty decent amount of traffic (around 300,000 pageviews a day). I am using BookSleeve for the redis connection and after the application gets up and going it will start throwing SocketExceptions from BookSleeve about four times a minute. The exact exception is:

Exception type: System.Net.Sockets.SocketException
Exception message: A connection attempt failed because the connected party did not     properly respond after a period of time, or established connection failed because connected host has failed to respond

It seems to only be happening when I am reading something from the server:

using (var connection = ConnectionGateway.GetReadConnection())
{
    var task = connection.Hashes.GetString(App.RedisDatabase, CacheKeys.My_KEY);
    var result = connection.Wait(task);
}

My GetReadConnection is setup like this:

    public static RedisConnection GetReadConnection()
    {
        RedisConnection conn = getNewConnection();

        try
        {
            var openAsync = conn.Open();
            conn.Wait(openAsync);

            return conn;
        }
        catch (SocketException ex)
        {
            throw new Exception(RedisConnectionFailed, ex);
        }
    }

Now all my writes share a single connection like the author describes so this is only happening on the reads which require the use of connection.Wait(). That all seems to work fine. The writes use code similar to Maintaining an open Redis connection using BookSleeve

I have tried changing the timeouts of the Redis server to account for the Azure load balancer with no success. I have tried both set timeout 30 and set timeout 0 as described here: Redis connection errors when using Booksleeve Redis client in Azure VM

Any help would really be appreciated.

Are you opening a connection per call? That could be very relevant... — Marc Gravell, Oct 05 '12 at 20:24
I thought it was mentioned that if you are not using wait then you make one massively shared connection. If you ARE using wait however you need to make a connection per call. So all my writes to redis are all using one shared connection, but the reads since they use wait instantiate a connection per call with the using code above. — Matthew Kruskamp, Oct 05 '12 at 20:28
no, that is simply not the case at all. When you use Wait, it carries on multiplexing all the other threads. All that happens is the thread that wants a result pauses until an answer *to that request* comes back - it is still concurrently processing all the others. That is pretty much the key design approach. For stackoverflow, we get away with a single connection for **all** our use. The main thing that causes problems for the multiplexer is transactions. My advice: try putting the reads through the shared connection - see if the problem goes away. — Marc Gravell, Oct 05 '12 at 20:43
Very interesting. I am going to try that and deploy it into production and see if it sorts out the issues. Thank you for the quick responses. — Matthew Kruskamp, Oct 05 '12 at 20:49
my main thought is that saturating both .net and redis with constant connections could cause a significant issue — Marc Gravell, Oct 05 '12 at 20:54
Thank you for all your help. We are deploying right now so I will have an update in a second. While I have the arbiter of redis in c# chatting with me I was wondering if http://stackoverflow.com/questions/8645953/maintaining-an-open-redis-connection-using-booksleeve is a solid way to maintain the shared connection. — Matthew Kruskamp, Oct 05 '12 at 21:15
if the redis server has a timeout, BookSleeve will add its own heartbeat. I could probably make that heartbeat code more accessible, to electively enable it? — Marc Gravell, Oct 05 '12 at 21:50
30 minutes so far and no problems. If you post the answer I will accept it for you. Thank you for your help. — Matthew Kruskamp, Oct 05 '12 at 22:22

score 3 · Accepted Answer · answered Oct 06 '12 at 06:26

Glancing at the code, the first thing that occurs is that you seem to be creating a connection per read operation. This is not required, and could be adversely impacting both .net and redis: BookSleeve is a multiplexer - it is designed to process lots of requests concurrently over a single connection. When you call Wait, it is only the waiting thread that blocks - the internals of BookSleeve will carry on handling other threads etc.

Additionally, creating a connection has quite a lot of overhead: in addition to the costs of establishing a new TCP connection, BookSleeve needs to talk to the redis server to find some key configuration information that it needs for correct operation, so it is always recommended to re-use a single shared connection or a small number of shared connections.

Connecting to Redis from Azure throwing Intermittent Exceptions

1 Answers1