Redis wildcard delete

Question

I'm using the the stackexchange redis provider and I'm trying to delete all keys starting with a certain key.

private readonly Lazy<IEnumerable<IServer>> allServers = new Lazy<IEnumerable<IServer>>(() => Connection.GetEndPoints().Select(s => Connection.GetServer(s)));
private const int DbId = 1;
private const int ScanPageSize = 1 << 14;

public void RemoveByPattern(string pattern)
{
    var keys = this.AllServers.Value.SelectMany(x => x.Keys(DbId, pattern + "*", ScanPageSize)).Distinct().ToArray();
    if (!keys.Any())
        return;
    this.Database.KeyDelete(keys);
}

^^ This is the method I've come up which works fine most of the time. But as soon as I put a bit of load on the system I'm getting errors that look like this:

  System.TimeoutException: Timeout performing SCAN, inst: 0, mgr: Inactive, queue: 11, qu=11, qs=0, qc=0, wr=0/1, in=0/0
     at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl (StackExchange.Redis.StrongName, Version=1.0.316.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46)
     at StackExchange.Redis.RedisServer.ExecuteSync (StackExchange.Redis.StrongName, Version=1.0.316.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46)
     at StackExchange.Redis.RedisBase+CursorEnumerable`1.GetNextPageSync (StackExchange.Redis.StrongName, Version=1.0.316.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46)
     at StackExchange.Redis.RedisBase+CursorEnumerable`1+CursorEnumerator.MoveNext (StackExchange.Redis.StrongName, Version=1.0.316.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46)
     at System.Linq.Enumerable+<SelectManyIterator>d__1`2.MoveNext (System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
     at System.Linq.Enumerable+<DistinctIterator>d__1`1.MoveNext (System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
     at System.Linq.Buffer`1..ctor (System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
     at System.Linq.Enumerable.ToArray (System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
     at Test.RedisCacheManager.RemoveByPattern (TestApplication, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)

Is there a better way of doing this? I know that if I knew about all the keys on the .net side i could avoid the scan. But unfortunately that's not particularly easy in my case.

Also note your code is not cluster-compatible. You should consider using a tagging mechanism to maintain the relation between the keys, like in [CachingFramework.Redis](https://github.com/thepirat000/CachingFramework.Redis) — thepirat000, Sep 25 '15 at 14:44
Please check my asnwer [here](http://stackoverflow.com/questions/27094938/azure-cache-datacache-style-regions-in-redis/34414377#34414377) to delete keys by wildcard — Kobynet, Mar 01 '16 at 14:19
Possible duplicate of [Redis Stack Exchange how to delete or get keys by pattern](http://stackoverflow.com/questions/26488830/redis-stack-exchange-how-to-delete-or-get-keys-by-pattern) — Paul Roub, Apr 14 '16 at 19:18

score 1 · Answer 1 · edited May 23 '17 at 12:24

I answered a similar question here. You can use the following with StackExchange.Redis.StrongName v1.0.488.

        foreach (var ep in _muxer.GetEndPoints())
        {
            var server = _muxer.GetServer(ep);
            var keys = server.Keys(database: _redisDatabase, pattern: pattern + "*").ToArray();
            _db.KeyDeleteAsync(keys);
        }

_muxer is instance of ConnectionMultiplexer

score 0 · Answer 2 · answered Sep 25 '15 at 07:43

0

Instead of removing by pattern you should be able to store partial keys or involved keys in some set and then scan it (i.e. sscan) using pages to drop the global keyspace keys.

Don't try to remove thousands of keys in a single pass: remember Redis is a single-threaded solution, and your removal will block the entire Redis server.

answered Sep 25 '15 at 07:43

Matías Fidemraizer

63,804
18
124
206

Any idea on how about doing that in c#? – Sam7 Sep 25 '15 at 08:11
@Sam7 Whenever you add a key that you want to remove later in the global space, you create a set using `IDatabase.SetAdd` or `IDatabaseAsync.SetAddAsync` – Matías Fidemraizer Sep 25 '15 at 08:18

score 0 · Answer 3 · answered Sep 25 '15 at 15:26

There are several options to do what you want, but I agree with @Matías in that you need to store the keys you want to remove later in a SET.

If you are running a standalone Redis the most reliable way will be to do the Add/Remove operations in a Lua Script. But if you plan to have a Redis cluster, you will have to do it client-side.

For example:

public void SetValue(string key, string value)
{
    var batch = db.CreateBatch(); // You can use a transaction with CreateTransaction() if you are not in a cluster.
    batch.StringSetAsync(key, value);
    batch.SetAddAsync("to remove", key);
    batch.Execute();
}
public void RemoveAll()
{
    var item = db.SetPop("to remove");
    while (item.HasValue)
    {
        db.KeyDelete(item.ToString());
        item = db.SetPop("to remove");
    }
}

Redis wildcard delete

3 Answers3