StackExchange.Redis with Azure Redis is unusably slow or throws timeout errors

Question

I'm moving all of my existing Azure In-Role cache use to Redis and decided to use the Azure Redis preview along with the StackExchange.Redis library (https://github.com/StackExchange/StackExchange.Redis). I wrote all the code for it without much problem, but when running it is absolutely unusably slow and constantly throws timeout errors (my timeout period is set to 15 seconds).

Here is the relevant code for how I am setting up the Redis connection and using it for simple operations:

    private static ConnectionMultiplexer _cacheService;
    private static IDatabase _database;
    private static object _lock = new object();

    private void Initialize()
    {
        if (_cacheService == null)
        {
            lock (_lock)
            {
                if (_cacheService == null)
                {
                    var options = new ConfigurationOptions();
                    options.EndPoints.Add("{my url}", 6380);
                    options.Ssl = true;
                    options.Password = "my password";
                    // needed for FLUSHDB command
                    options.AllowAdmin = true;

                    // necessary?
                    options.KeepAlive = 30;
                    options.ConnectTimeout = 15000;
                    options.SyncTimeout = 15000;

                    int database = 0;

                    _cacheService = ConnectionMultiplexer.Connect(options);
                    _database = _cacheService.GetDatabase(database);
                }
            }
        }

    }

    public void Set(string key, object data, TimeSpan? expiry = null)
    {
        if (_database != null)
        {
            _database.Set(key, data, expiry: expiry);
        }
    }

    public object Get(string key)
    {
        if (_database != null)
        {
            return _database.Get(key);
        }
        return null;
    }

Performing very simple commands like Get and Set often time out or take 5-10 seconds to complete. Seems like it kind of negates the whole purpose of using it as a cache if it's WAY slower than actually fetching the real data from my database :)

Am I doing anything obviously incorrect?

Edit: here are some stats that I pulled from the server (using Redis Desktop Manager) in case that sheds some light on anything.

Server
redis_version:2.8.12
redis_mode:standalone
os:Windows  
arch_bits:64
multiplexing_api:winsock_IOCP
gcc_version:0.0.0
process_id:2876

tcp_port:6379
uptime_in_seconds:109909
uptime_in_days:1
hz:10
lru_clock:16072421
config_file:C:\Resources\directory\xxxx.Kernel.localStore\1\redis_2092_port6379.conf

Clients
connected_clients:5
client_longest_output_list:0
client_biggest_input_buf:0
client_total_writes_outstanding:0
client_total_sent_bytes_outstanding:0
blocked_clients:0

Memory
used_memory:4256488
used_memory_human:4.06M
used_memory_rss:67108864
used_memory_rss_human:64.00M
used_memory_peak:5469760
used_memory_peak_human:5.22M
used_memory_lua:33792
mem_fragmentation_ratio:15.77
mem_allocator:dlmalloc-2.8

Persistence
loading:0
rdb_changes_since_last_save:72465
rdb_bgsave_in_progress:0
rdb_last_save_time:1408471440
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

Stats
total_connections_received:25266
total_commands_processed:123389
instantaneous_ops_per_sec:10
bytes_received_per_sec:275
bytes_sent_per_sec:65
bytes_received_per_sec_human:

Edit 2: Here are the extension methods I'm using for Get/Set -- they are very simple methods that just turn an object into JSON and call StringSet.

    public static object Get(this IDatabase cache, string key)
    {
        return DeserializeJson<object>(cache.StringGet(key));
    }

    public static void Set(this IDatabase cache, string key, object value, TimeSpan? expiry = null)
    {
        cache.StringSet(key, SerializeJson(value), expiry: expiry);
    }

Edit 3: here are a couple example error messages:

    A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll
    Timeout performing GET MyCachedList, inst: 11, queue: 1, qu=1, qs=0, qc=0, wr=0/1, in=0/0

    A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll
    Timeout performing GET MyCachedList, inst: 1, queue: 97, qu=0, qs=97, qc=0, wr=0/0, in=3568/0

There are no `Get` or `Set` methods on `StackExchange.Redis.IDatabase`. Are you using extension methods? If so, can you post the code for those? — Mark Rendle, Aug 21 '14 at 13:04
@MarkRendle I've added my Get/Set extension methods. It's pretty simple. I've excluded the SerializeJson/DeserializeJson methods, but if you're curious they use the Newtonsoft library. — user2719100, Aug 21 '14 at 15:19
You are not alone, we've been struggling with this and have decided to just move our redis instance to a VM. Tired of it... — Jeff Borden, Aug 21 '14 at 16:54
@JeffBorden Good to know I'm not the only one. I may just move it all to ElastiCache instead — user2719100, Aug 25 '14 at 15:35
@user2719100 Yea, moved our cloud service and redis instance to a VM and haven't been happier. I stuck with Azure way too long... — Jeff Borden, Aug 25 '14 at 15:41
@user2719100 I am having similar issue, could you please have a look at my question? http://stackoverflow.com/questions/29569997/azure-redis-cache-multiple-errors-timeoutexception-timeout-performing-get-ke — Jakub Holovsky, Apr 13 '15 at 08:21
Can you share what size cache you are using? If you are using a 250 MB cache, could you try using the 1GB option and see if that helps. Also if you would be willing to share your cache name, I can look at our server logs to see if that can shed any light on the high latency you are seeing. Before this thread I had not heard of performance complains with Azure Redis Cache. Will investigate more at our end and get back to you ... — Saurabh, Aug 22 '14 at 23:27

score 25 · Answer 1 · edited Sep 22 '16 at 19:34

25

Here is the recommended pattern, from the Azure Redis Cache documentation:

private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => {
    return ConnectionMultiplexer.Connect("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");
});

public static ConnectionMultiplexer Connection {
    get {
        return lazyConnection.Value;
    }
}

A few important points:

It uses Lazy<T> to handle thread-safe initialization
It sets "abortConnect=false", which means if the initial connect attempt fails, the ConnectionMultiplexer will silently retry in the background rather than throw an exception.
It does not check the IsConnected property, since ConnectionMultiplexer will automatically retry in the background if the connection is dropped.

edited Sep 22 '16 at 19:34

Sean

8,407
3
31
33

answered Mar 02 '15 at 23:24

Mike Harder

1,765
16
12

And the IDatabase? Can you put a static field with the `IDatabase` and only initialize it on start with `Connection.GetDatabase()` ? – Dirk Boer Feb 13 '16 at 12:40
2

@DirkBoer, per the documentation (https://github.com/StackExchange/StackExchange.Redis/blob/master/Docs/Basics.md#using-a-redis-database), "The object returned from GetDatabase is a cheap pass-thru object, and does not need to be stored." I think the intended use of the database object is to get a new instance for every operation. – Sean Sep 22 '16 at 18:45
Is this applied to nodejs as well? – Enxhi Leba Jun 20 '23 at 12:50

score 5 · Answer 2 · answered Mar 18 '15 at 18:14

I was having similar issues. Redis cache was unusually slow but was definitely caching. In some cases, it took 20-40 seconds to load a page.

I realized that the cache server was in a different location than the site's. I updated the cache server to live in the same location as the website and now everything works as expected.

That same page now loads in 4-6 seconds.

Good luck to anyone else who's having these issues.

Sharat Pandavula · Answer 3 · 2014-09-23T13:42:31.190

3

The problem is how the connection object created and used. we faced exact problem initially and fixed with a single connection object getting used across all web requests. And we check is it null or connected in session start for graceful re creating object. that fixed the issue.

Note: Also check in which Zone of Azure your Redis Cache instance is running and Which Zone your Web Server exist. It is better to maintain both in Same Zone

In Global.ascx.cs file

public static ConnectionMultiplexer RedisConnection;
public static IDatabase RedisCacheDb;

protected void Session_Start(object sender, EventArgs e)
    {
        if (ConfigurationManager.ConnectionStrings["RedisCache"] != null)
        {
            if (RedisConnection == null || !RedisConnection.IsConnected)
            {
                RedisConnection = ConnectionMultiplexer.Connect(ConfigurationManager.ConnectionStrings["RedisCache"].ConnectionString);
            }
            RedisCacheDb = RedisConnection.GetDatabase();
        }
    }

edited Sep 23 '14 at 13:42

answered Sep 20 '14 at 08:19

Sharat Pandavula

72
4

Thanks for this response. I'm not quite sure how this is different than my original code since I am also using a static singleton. Since I posted this question I have completely migrated away from Azure over to AWS and this is no longer an issue. – user2719100 Sep 22 '14 at 16:40
static singleton is not same in all contexts, creating in Global.ascx is at application context. you can revisit your code, but Azure is not the issue. If interested check in which Azure Zone you created your Redis Cache instance and from which Zone you are using it! – Sharat Pandavula Sep 23 '14 at 13:27
2

Static, by definition in the CLR *is* always the same. It is an instance shared throughout an entire app domain -- it should not matter when/where/how it was instantiated. The Redis instance I set up at Azure was in the exact same zone as my application, and without changing any code I have moved to AWS I do not experience the issue. That's as much as I can tell you. You can claim the issue was not Azure, but that just makes you sound like you work for Microsoft :) – user2719100 Sep 23 '14 at 16:53
we both solved our issues, so let me not dig more. Hope this might help who are still facing this issue. I am not working for MS though, I am an Azure user only! – Sharat Pandavula Sep 24 '14 at 08:56
3

It seems a bit strange to do this in Session_Start. You don't need to do this for every new user (Session) that connects to the site. – gliljas Dec 02 '14 at 23:11
3

Another problem with this approach is that connections will be leaked until the ConnectionMultiplexer instances are cleaned up the CLR garbage collector. If the cache is under heavy load it's possible to hit the 10k connection limit. There's also no good way to call Dispose() on the old ConnectionMultiplexer, since another thread could still be trying to use it. The best approach we have found is to ignore the IsConnected property and let ConnectionMultiplexer retry the connection on its own. – Mike Harder Mar 02 '15 at 23:23
+1 @MikeHarder agreed, This recommendation was not available when I was using it at preview stage and abortConnect=false is part of our connectionstring. – Sharat Pandavula Mar 04 '15 at 12:17
Tried all that has been mentioned and its still kind of slow. When I replace the redis cache with a database table instead it is so much faster. Something is clearly not right as you use caching to be faster than the database. I'd expect 1000 calls to the cache to take around a second or two. 4-6 seconds to load a page is crazy slow - unless your doing some crazy computation. – Shumii Jan 26 '16 at 06:18

score 3 · Answer 4 · edited May 27 '20 at 04:34

3

In our case the issue is when using SSL connection. You're showing that your desktop manager is running on the non-SSL port, but your code is using SSL.

A quick benchmark on our Azure redis without SSL, retrieving around 80k values with an LRANGE command (also with .net and StackExchange.Redis) is basically instand. When SSL is used, the same query takes 27 seconds.

WebApp: Standard S2
Redis: Standard 1 GB

Edit: Checking the SLOWLOG, Redis itself seems to actually hit slowlog with it's 14ms time it takes or so to grab the rows, but this is far from the actual transfer with SSL enabled. We ended up with a premium Redis to have some sort of security between Redis and Web Apps.

edited May 27 '20 at 04:34

Community

1
1

answered Aug 17 '15 at 09:59

Crypth

1,576
18
32

Did you find that using non-ssl continued to solve your problems? I tried this and noticed no effect for my setup. – Michael Haren Nov 02 '15 at 14:18
Yes, using non-ssl speeds up this immensely. Both client and server resided in the same location. Tried this just recently again and the problem is still there. – Crypth Nov 13 '15 at 14:36
With have our custom setup of Redis on VM and have the same problem. Have you figured out real reasons of slow performance? – alexey Jul 15 '17 at 10:50
Afraid not, as i wrote in my answer, ended up with a premium Redis in a VLAN and run the connection between webapps and Redis unsecure. – Crypth Jul 31 '17 at 14:38

score 3 · Answer 5 · answered May 28 '16 at 13:09

It worked in my case. Don't forget to increase the SyncTimeout. The default is 1 second.

private static Lazy<ConnectionMultiplexer> ConnectionMultiplexerItem = new Lazy<ConnectionMultiplexer>(() =>
{
    var redisConfig = ConfigurationOptions.Parse("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");

    redisConfig.SyncTimeout = 3000;

    return ConnectionMultiplexer.Connect(redisConfig);
});

Check if you your Azure Redis Cache and the Client in the same region in Azure. For example, you might be getting timeouts when your cache is in East US but the client is in West US and the request doesn't complete in synctimeout time or you might be getting timeouts when you are debugging from your local development machinex. It’s highly recommended to have the cache and in the client in the same Azure region. If you have a scenario to do a cross region calls, you would want to set the synctimeout to a higher value.

Read more: https://azure.microsoft.com/en-us/blog/investigating-timeout-exceptions-in-stackexchange-redis-for-azure-redis-cache/

StackExchange.Redis with Azure Redis is unusably slow or throws timeout errors

5 Answers5

Linked