Classical approaches and quotations
From msdn, by Stephen Cleary
Asynchronous code is often used to initialize a resource that’s then
cached and shared. There isn’t a built-in type for this, but Stephen
Toub developed an AsyncLazy that acts like a merge of Task and
Lazy. The original type is described on his blog, and an
updated version is available in my AsyncEx library.
public class AsyncLazy<T> : Lazy<Task<T>>
{
public AsyncLazy(Func<T> valueFactory) :
base(() => Task.Factory.StartNew(valueFactory)) { }
public AsyncLazy(Func<Task<T>> taskFactory) :
base(() => Task.Factory.StartNew(() => taskFactory()).Unwrap()) { }
}
Context
Let’s say in our program we have one of these AsyncLazy instances:
static string LoadString() { … }
static AsyncLazy<string> m_data = new AsyncLazy<string>(LoadString);
Usage
Thus, we can write an asynchronous method that does:
string data = await m_data.Value;
The Lazy<T> would be appropriate, but unfortunately it seems to lack the input parameter to index the result. The same issue was solved here where it is explained how to cache the results from a long-running, resource-intensive method, in case it is not async
Back to your proposed solution
Before I show the main changes related to the cache management and specific to your proposed implementation, let me suggest a couple of marginal optimization options, based on the following concerns.
often with locks, when you access them they’re uncontended, and in
such cases you really want acquiring and releasing the lock to be as
low-overhead as possible; in other words, accessing uncontended locks
should involve a fast path
Since they're just performance optimization tricks, I will leave them commented in the code so that you can measure their effects in your specific situation before.
- You need to test TryGetValue again after awaiting because another parallel process could have added that value in the meantime
- You don't need to keep the lock while you're awaiting
This balance of overhead vs cache misses was already pointed out in a previous answer to a similar question.
Obviously, there's overhead keeping SemaphoreSlim objects around to
prevent cache misses so it may not be worth it depending on the use
case. But if guaranteeing no cache misses is important than this
accomplishes that.
My main answer: the cache management
Regarding the cache expiration, I would suggest to add the creation DateTime to the value of the Dictionary (i.e. the time when the value is returned from GetSomethingTheLongWayAsync) and consequently discard the cached value after a fixed time span.
Find a draft below
private static readonly ConcurrentDictionary<object, SemaphoreSlim> _keyLocks = new ConcurrentDictionary<object, SemaphoreSlim>();
private static readonly ConcurrentDictionary<object, Tuple<string, DateTime>> _cache = new ConcurrentDictionary<object, Tuple<string, DateTime>>();
private static bool IsExpiredDelete(Tuple<string, DateTime> value, string key)
{
bool _is_exp = (DateTime.Now - value.Item2).TotalMinutes > Expiration;
if (_is_exp)
{
_cache.TryRemove(key, out value);
}
return _is_exp;
}
public async Task<string> GetSomethingAsync(string key)
{
Tuple<string, DateTime> cached;
// get the semaphore specific to this key
var keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
await keyLock.WaitAsync();
try
{
// try to get value from cache
if (!_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached,key))
{
//possible performance optimization: measure it before uncommenting
//keyLock.Release();
string value = await GetSomethingTheLongWayAsync(key);
DateTime creation = DateTime.Now;
// in case of performance optimization
// get the semaphore specific to this key
//keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
//await keyLock.WaitAsync();
bool notFound;
if (notFound = !_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached, key))
{
cached = new Tuple<string, DateTime>(value, creation);
_cache.TryAdd(key, cached);
}
else
{
if (!notFound && cached.Item2 < creation)
{
cached = new Tuple<string, DateTime>(value, creation);
_cache.TryAdd(key, cached);
}
}
}
}
finally
{
keyLock.Release();
}
return cached?.Item1;
}
Please, adapt the above code to your specific needs.
Making it more generic
Finally you may want to generalize it a little bit.
By the way, notice that the Dictionary are not static since one could cache two different methods with the same signature.
public class Cached<FromT, ToT>
{
private Func<FromT, Task<ToT>> GetSomethingTheLongWayAsync;
public Cached (Func<FromT, Task<ToT>> _GetSomethingTheLongWayAsync, int expiration_min ) {
GetSomethingTheLongWayAsync = _GetSomethingTheLongWayAsync;
Expiration = expiration_min;
}
int Expiration = 1;
private ConcurrentDictionary<FromT, SemaphoreSlim> _keyLocks = new ConcurrentDictionary<FromT, SemaphoreSlim>();
private ConcurrentDictionary<FromT, Tuple<ToT, DateTime>> _cache = new ConcurrentDictionary<FromT, Tuple<ToT, DateTime>>();
private bool IsExpiredDelete(Tuple<ToT, DateTime> value, FromT key)
{
bool _is_exp = (DateTime.Now - value.Item2).TotalMinutes > Expiration;
if (_is_exp)
{
_cache.TryRemove(key, out value);
}
return _is_exp;
}
public async Task<ToT> GetSomethingAsync(FromT key)
{
Tuple<ToT, DateTime> cached;
// get the semaphore specific to this key
var keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
await keyLock.WaitAsync();
try
{
// try to get value from cache
if (!_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached, key))
{
//possible performance optimization: measure it before uncommenting
//keyLock.Release();
ToT value = await GetSomethingTheLongWayAsync(key);
DateTime creation = DateTime.Now;
// in case of performance optimization
// get the semaphore specific to this key
//keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
//await keyLock.WaitAsync();
bool notFound;
if (notFound = !_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached, key))
{
cached = new Tuple<ToT, DateTime>(value, creation);
_cache.TryAdd(key, cached);
}
else
{
if (!notFound && cached.Item2 < creation)
{
cached = new Tuple<ToT, DateTime>(value, creation);
_cache.TryAdd(key, cached);
}
}
}
}
finally
{
keyLock.Release();
}
return cached.Item1;
}
}
For a generic FromT an IEqualityComparer is needed for the Dictionary
Usage/Demo
private static async Task<string> GetSomethingTheLongWayAsync(int key)
{
await Task.Delay(15000);
Console.WriteLine("Long way for: " + key);
return key.ToString();
}
static void Main(string[] args)
{
Test().Wait();
}
private static async Task Test()
{
int key;
string val;
key = 1;
var cache = new Cached<int, string>(GetSomethingTheLongWayAsync, 1);
Console.WriteLine("getting " + key);
val = await cache.GetSomethingAsync(key);
Console.WriteLine("getting " + key + " resulted in " + val);
Console.WriteLine("getting " + key);
val = await cache.GetSomethingAsync(key);
Console.WriteLine("getting " + key + " resulted in " + val);
await Task.Delay(65000);
Console.WriteLine("getting " + key);
val = await cache.GetSomethingAsync(key);
Console.WriteLine("getting " + key + " resulted in " + val);
Console.ReadKey();
}
Sophisticated alternatives
There are also more advanced possibilities like the overload of GetOrAdd that takes a delegate and Lazy objects to ensure that a generator function is called only once (instead of semaphores and locks).
public class AsyncCache<FromT, ToT>
{
private Func<FromT, Task<ToT>> GetSomethingTheLongWayAsync;
public AsyncCache(Func<FromT, Task<ToT>> _GetSomethingTheLongWayAsync, int expiration_min)
{
GetSomethingTheLongWayAsync = _GetSomethingTheLongWayAsync;
Expiration = expiration_min;
}
int Expiration;
private ConcurrentDictionary<FromT, Tuple<Lazy<Task<ToT>>, DateTime>> _cache =
new ConcurrentDictionary<FromT, Tuple<Lazy<Task<ToT>>, DateTime>>();
private bool IsExpiredDelete(Tuple<Lazy<Task<ToT>>, DateTime> value, FromT key)
{
bool _is_exp = (DateTime.Now - value.Item2).TotalMinutes > Expiration;
if (_is_exp)
{
_cache.TryRemove(key, out value);
}
return _is_exp;
}
public async Task<ToT> GetSomethingAsync(FromT key)
{
var res = _cache.AddOrUpdate(key,
t => new Tuple<Lazy<Task<ToT>>, DateTime>(new Lazy<Task<ToT>>(
() => GetSomethingTheLongWayAsync(key)
)
, DateTime.Now) ,
(k,t) =>
{
if (IsExpiredDelete(t, k))
{
return new Tuple<Lazy<Task<ToT>>, DateTime>(new Lazy<Task<ToT>>(
() => GetSomethingTheLongWayAsync(k)
), DateTime.Now);
}
return t;
}
);
return await res.Item1.Value;
}
}
Same usage, just replace AsyncCache instead of Cached.
