StackExchange.Redis ConnectionMultiplexer.Connect() Intermittently Works

Question

I am using StackExchange.Redis to talk to 3 different Redis instances: 1 on the same subnet and 2 remotely. Here's my configuration code:

var configurationOptions = new ConfigurationOptions
{
    EndPoints =
    {
        { host, port }
    },
    KeepAlive = 180,
    Password = password,
    DefaultVersion = new Version("2.8.5"),
    // Needed for cache clear
    AllowAdmin = true
};

var connectionMultiplexer = ConnectionMultiplexer.Connect(configurationOptions );

the last line throws a connection exception approximately 70% of the time:

It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail

Why is this intermittent and/or what am I doing wrong? When I ping the Redis server in a command prompt, there is 0% packet loss and a <1 ms response. The network is stable.

Thanks!

EDIT

Here is what the log outputs when it fails:

10.48.68.28:6379,keepAlive=180,version=2.8.5

1 unique nodes specified
Requesting tie-break from 10.48.68.28:6379 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:01 to respond...
10.48.68.28:6379 did not respond
10.48.68.28:6379 failed to nominate (WaitingForActivation)
No masters detected
10.48.68.28:6379: Standalone v2.8.5, master; keep-alive: 00:03:00; int: Connecting; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
10.48.68.28:6379: int ops=0, qu=4, qs=0, qc=0, wr=0, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, subs=1, sync=2, socks=1
Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
Starting heartbeat...

If you pass in a TextWriter as the log parameter (StringWriter works fine) - what does it say? — Marc Gravell, Apr 18 '14 at 19:48
LSO: is this right at first startup? First time? I'm wondering how much is JIT / Fusion / DNS / etc — Marc Gravell, Apr 18 '14 at 19:48
Right at startup, first time. When I expand the sync timeout to 30000ms it doesn't help. Always waits the max and fails. Trying `BookSleeve` right now to see if it works. — Haney, Apr 18 '14 at 19:53
Curious. Very curious. Question: are you specifying an ip? Or a host name? — Marc Gravell, Apr 18 '14 at 19:56
IP for 1 of the instances, hostname for the other 2. I am trying the log now and will report back in a sec. Also a given process will only be configured for 1 of the 3 instances (think DEV, QA, PROD) — Haney, Apr 18 '14 at 19:57
the qu=4 is **very** interesting to me - it suggests the writer didn't start. Can you tell me what version you are using exactly? Basically qu is the unsent queue: there are 4 messages that for some reason have not yet been sent. This troubles me. — Marc Gravell, Apr 18 '14 at 21:21
The fact that it is connecting rather than connection established also intriguing — Marc Gravell, Apr 18 '14 at 21:23
@MarcGravell running [latest nuget package](http://www.nuget.org/packages/StackExchange.Redis) 1.0.270 — Haney, Apr 18 '14 at 23:34
@MarcGravell if I can help by testing in any way, let me know. Happy to do so. Just wanna make it work. :) — Haney, Apr 19 '14 at 15:22
long weekend here: I'll have to look when i can. Does increasing the connect timeout help? — Marc Gravell, Apr 19 '14 at 15:24
Nope, if I increase the timeout to 60 seconds, it takes 60 seconds to fail. Which would be in line with your "it isn't sending because it's queued" thing. Enjoy the long weekend! We shall reconvene next week. :) — Haney, Apr 19 '14 at 15:58
@Craig to the best of my knowledge, Marc is still looking into it. If there's a new version you could try upgrading to it. I went back to BookSleeve for the time being and it worked well. — Haney, May 21 '14 at 16:58

score 10 · Answer 1 · answered Nov 25 '14 at 23:27

10

I was able to workaround it by setting a ConnectTimeouton the client when connecting to Redis. Here was my code

 ConnectionMultiplexer connection = 
        ConnectionMultiplexer.Connect("endpoint,password=password,ConnectTimeout=10000");

answered Nov 25 '14 at 23:27

pranav rastogi

4,124
23
23

score 6 · Answer 2 · answered May 29 '14 at 22:07

6

This seems to be a somewhat common issue: https://github.com/StackExchange/StackExchange.Redis/issues/42

I just downloaded the new build and haven't seen the problem again. Yet.

answered May 29 '14 at 22:07

shawn.r.stevens

61
2

score 5 · Answer 3 · edited Jun 06 '23 at 13:14

The error you are getting is usually a sign that you have not set abortConnect=false in your connection string. The default value for abortConnect is true, which makes it so that StackExchange.Redis won't reconnect to the server automatically under some conditions. We strongly recommend that you set abortConnect=false in your connection string so that Redis will auto-reconnect in the background if a network blip occurs.

Source: https://stackoverflow.com/a/30918632/2236811

So my init() looked like this

ConfigurationOptions co = new ConfigurationOptions()
{
    SyncTimeout = 500000,
    EndPoints =
    {
        {url,portNumber }
    },
    AbortOnConnectFail = false // this prevents that error
};

seClient = ConnectionMultiplexer.Connect(co);

Thanks

score 3 · Answer 4 · answered Dec 12 '14 at 19:27

For me this error was because total number of connection reached to its max limit i.e. 10K. Restarting Client and kill some of them solved the problem. You can check Azure portal and figure out if you have reached max limit for any of the resources.

StackExchange.Redis ConnectionMultiplexer.Connect() Intermittently Works

4 Answers4