Booksleeve (1.3.39) + Redis 2.6.16 + Sentinel: The connection has been closed (Error)

Question

I am getting some random disconnects from Booksleeve when deployed in the production environment. I have run tests on my machine and I can not seem to reproduce the errors. I am not sure what to check so I decided to post here.

My set-up:

2 x Redis 2.6.16 (Debian Linux)

2 x Redis sentinel (running on the same machines as redis)

4 x Windows Server 2008 R2

Application connection handling

My app has one static connection that is opened using ConnectionUtils. I also have a timer (System.Timers.Timer) that runs every connection.Ping every 5'' to check whether the connection is still open. If I get an exception for the Ping I log it, close and dispose the connection and replace it with a new one using ConnectionUtils again. I am also logging Task.UnobservedExceptions

Here is what I am getting in my logs:

A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
   at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
   at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
   at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
   at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
   at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
   at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
   at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
   at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
   at System.Threading.Tasks.Task.Execute()

And this is my connection monitor reporting on the error in the ping:

Redis connection is broken: System.InvalidOperationException: The connection has been closed (Error); no new messages can be delivered; the last command was sent 1453ms ago
   at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
   at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
   at BookSleeve.RedisConnectionBase.PingImpl(Boolean queueJump, Boolean duringInit, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 129
   at Atcom.Data.Caching.RedisConnectionContainer.CheckConnections(Object sender, EventArgs e) in C:\Projects\Atcom.Data.Caching.Redis\Atcom.Data.Caching.Redis\RedisConnectionContainer.cs:line 66

I have tried connecting through both the Sentinels and directly to Redis but I am getting the same thing.

I have asked my network administrator to check but he says there is no problem (and I cant really argue with what I dont know oO)

I have reviewed Redis settings and documentation and as far as I can tell clients that do not use monitor etc. should not get disconnected by redis. Idle time disconnect is also disabled on redis.

Could sentinel (since its beta) be the culprit?

Any help troubleshooting the problem would be GREATLY appreciated.

Not sure about fixing the core problem (hence this is a comment and not an answer) but have a look at [this answer](http://stackoverflow.com/a/8777999/290343) that demonstrates a class that you can use to connect to BookSleeve. Internally it takes care of re-instantiating dead connections. — Ofer Zelig, Nov 07 '13 at 23:01
I don't really like those locks oO. I made a decision a while back that I would sacrifice some reliability for better performance. That is why I check if the connection is open every X seconds instead of every time I am using it. Besides, I dont have multiple connections spawning. — John, Nov 08 '13 at 10:11

Booksleeve (1.3.39) + Redis 2.6.16 + Sentinel: The connection has been closed (Error)

0 Answers0