5

Caveat: Okay so this is a weird one, and i'm not sure if SO is the right place.

I have an Azure Website connecting to an Azure Redis Cache instance. (using StackExchange.Redis)

Everything was great, then one day - the website couln't connect to Redis.

Error:

It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING

Here's my connection string:

mycache.redis.cache.windows.net,ssl=true,password=xxxxxx,syncTimeout=5000

Here were my diagnosis steps:

  1. Try and connect from local to Azure Redis. Result: SUCCESS (so code is good?)
  2. Try and spinup NEW Azure Redis instance, connect from Azure. Result: FAIL (website can't connect to ANY azure Redis instance?)
  3. Spinup NEW Azure Website, with same code as erroring code, pointing to existing Redis cache. Result: SUCCESS (um, what?)
  4. File new MVC website, add StackExchange.Redis, deploy to new Azure Website, connecting to Redis. Result: SUCCESS (so Redis is good?)
  5. Deploy above vanilla MVC website to existing Azure Website (so same code as 4, connecting to same Redis, only difference is it's using the old Azure Website physical machine/networking). Result: FAIL (wtf??)

So - i'm thinking Redis has "blacklisted" the Azure website? (is that even possible?) I know that the client (my code) won't try and keep reconnecting, but i've bounced the site many times, and it just can't reconnect to Redis.

The fact that spinning up a new Azure Website, with the same code connecting to the same Redis instance results in success, tells me that some kind of blacklisting/routing issue has occured in Azure/Redis.

Any ideas?

EDIT

Looks like the problem is Azure VNET. When my website is part of the Azure Virtual Network, it can't connect to Redis. But when i take it out of the network, it connects fine. Before today, this setup was working fine.

So im wondering if Azure has made a change so that websites in a VNET cannot connect to Azure Redis? (makes no sense i know)

EDIT 2:

Attached is the logs from the Redis connection attempt.

Exception: It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail. SocketFailure on PING connection-string-removed:6380,password=password-removed,ssl=True Connecting connection-string-removed:6380/Interactive... BeginConnect: connection-string-removed:6380 1 unique nodes specified Requesting tie-break from connection-string-removed:6380

__Booksleeve_TieBreak... Allowing endpoints 00:00:05 to respond... Awaiting task completion, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=4,Free=32763,Min=1,Max=32767) Not all tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=5,Free=32762,Min=1,Max=32767) connection-string-removed:6380 did not respond Awaiting task completion, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=5,Free=32762,Min=1,Max=32767) Not all tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=5,Free=32762,Min=1,Max=32767) connection-string-removed:6380 failed to nominate (WaitingForActivation) No masters detected connection-string-removed:6380: Standalone v2.0.0, master; keep-alive: 00:01:00; int: Connecting; sub: Connecting; not in use: DidNotRespond connection-string-removed:6380: int ops=0, qu=2, qs=0, qc=0, wr=0, socks=1; sub ops=0, qu=0, qs=0, qc=0, wr=0, socks=1 Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0 (0.00 ops/s; spans 10s) Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago resetting failing connections to retry... retrying; attempts left: 2... 1 unique nodes specified Requesting tie-break from connection-string-removed:6380 > __Booksleeve_TieBreak... Allowing endpoints 00:00:05 to respond... Awaiting task completion, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=6,Free=32761,Min=1,Max=32767) Not all tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=7,Free=32760,Min=1,Max=32767) connection-string-removed:6380 did not respond Awaiting task completion, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=7,Free=32760,Min=1,Max=32767) Not all tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=7,Free=32760,Min=1,Max=32767) connection-string-removed:6380 failed to nominate (WaitingForActivation) No masters detected connection-string-removed:6380: Standalone v2.0.0, master; keep-alive: 00:01:00; int: Connecting; sub: Connecting; not in use: DidNotRespond connection-string-removed:6380: int ops=0, qu=2, qs=0, qc=0, wr=0, async=3, socks=2; sub ops=0, qu=0, qs=0, qc=0, wr=0, socks=2 Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0 (0.00 ops/s; spans 10s) Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago resetting failing connections to retry... retrying; attempts left: 1... 1 unique nodes specified Requesting tie-break from connection-string-removed:6380 > __Booksleeve_TieBreak... Allowing endpoints 00:00:05 to respond... Awaiting task completion, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=8,Free=32759,Min=1,Max=32767) EndConnect: connection-string-removed:6380 (socket shutdown) Connect complete: connection-string-removed:6380 All tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=11,Free=32756,Min=1,Max=32767) connection-string-removed:6380 faulted: SocketFailure on PING Awaiting task completion, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=11,Free=32756,Min=1,Max=32767) Not all tasks completed cleanly, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=7,Free=32760,Min=1,Max=32767) connection-string-removed:6380 failed to nominate (WaitingForActivation) No masters detected connection-string-removed:6380: Standalone v2.0.0, master; keep-alive: 00:01:00; int: Connecting; sub: Connecting; not in use: DidNotRespond connection-string-removed:6380: int ops=0, qu=2, qs=0, qc=0, wr=0, async=7, socks=3; sub ops=0, qu=0, qs=0, qc=0, wr=0, socks=3 Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0 (0.00 ops/s; spans 10s) Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago

Can anyone decipher this?

RPM1984
  • 72,246
  • 58
  • 225
  • 350
  • @marcgravell please be out there :) – RPM1984 Jun 18 '15 at 05:09
  • 1
    Azure Redis doesn't have any logic around VNET support or blacklisting sites, so the issue is unlikely on the Azure Redis side of things. Please send an e-mail to azurecache at Microsoft.com with the name of your cache along with the name of your website where you are seeing the failure and we will see if we can figure out what is going on... – JonCole Jun 18 '15 at 14:41
  • hi @JonCole thanks. :) i've emailed "azurecache@microsoft.com". It's interesting, as SOON as i take it out of the VNET, it works fine. Same goes for connecting to third-party Redis service (e.g RedisLabs). In VNET = fail, out of VNET = success. – RPM1984 Jun 18 '15 at 23:15
  • @JonCole - i haven't got any email back. Is azurecache@microsoft.com the right email? – RPM1984 Jun 21 '15 at 00:59
  • Yes, this is right. We will try to reply today. – JonCole Jun 22 '15 at 17:35
  • As a side note, please review this question as you seem to also have the same problem there in addition to the problem you are describing in this question: http://stackoverflow.com/questions/30895507/it-was-not-possible-to-connect-to-the-redis-servers-to-create-a-disconnected/30918632#30918632 – JonCole Jun 23 '15 at 00:19
  • Had the same problem with a new Premium Tier Redis cache today. Settled for a Standard Tier instance instead until this bug is fixed. – Rush Frisby Nov 22 '15 at 03:54

3 Answers3

2

I'm with the Azure Web Apps team - it looks like your VNET got into a particularly strange state, and was interrupting network connectivity for your app. I have fixed this behavior.

We are incredibly sorry for the inconvenience...

Aleks B
  • 71
  • 4
  • Hi Aleks, thanks - but can you please confirm a few things. 1) Which VNET/websites/Redis instances you are referring to (we are having this issue with multple envionrments), 2) What was the issue, 3) When did you fix the problem, 4) Is this related to the ticket i raised? (if so, can you please reply there, instead of SO) – RPM1984 Jun 24 '15 at 22:40
  • I was responding due to the email. I will provide more information to you in the support ticket. – Aleks B Jun 25 '15 at 16:16
0

We might be experiencing the same issue. I was able to put a test app onto a web app in Azure without a virtual network and it works right out-of-the-box (using Stackexchange.Redis). When I put the same code onto a web app that's part of a virtual network it doesn't work.

I managed to fix the first error ("It was not possible to connect to the redis server(s)..") by setting the AbortOnConnectFail to false. Then I got the error "No connection is available to service this operation: EXISTS foo" (= in this case trying to check if key 'foo' exists).

I am able to fix that too by setting ConnectTimeout to 10 seconds. So basically, I can get it to work, but that seems to cause long delays when (I assume) SE.Redis looses connection and tries to reconnect.

mikkark
  • 135
  • 9
  • Yep, `abortOnConnect` fixes the initial connection blip for me, but sometimes i still get connection issues. Network blips happen, and we just have to deal with that. – RPM1984 Jul 22 '15 at 23:12
  • Azure support helped us resolve this, it turned out the virtual network was incorrectly configured. It had two invalid DNS servers set, which caused the Redis problem (it also caused other problems, which were resolved). I removed the DNS servers and everything started working. – mikkark Aug 05 '15 at 06:12
0

If aboves posts did not help, you can check :

  • double check your password (private key) for correctness
  • try to check/uncheck "32-bit prefer" in project properties if you app is executable.
  • try to off ssl on from Azure redis side (from ui) and try set useSsl to false.
  • download source code (from there : https://github.com/StackExchange/StackExchange.Redis) and try to debug inner issue.

Part of that manipulation helped me.

Nigrimmist
  • 10,289
  • 4
  • 52
  • 53