20

I'm new to redis so I'm doing something wrong, I'm sure:

I've stored roughly 16,000 key/values in Azure Redis.

I used the following to write the keys/values

      foreach (var worksheet in wksList)
      {
        var wksYYMM = string.Format("{0}{1:00}", worksheet.ReviewDt.Year, worksheet.ReviewDt.Month);
        var wksKey = string.Format("{0}:{1}:{2}", provCode, wksYYMM, worksheet.AcctNbr);
        string jsonStr = JsonConvert.SerializeObject( MakeWsListDto(worksheet, provCoderList, rvrList));
        cache.StringSet(wksKey, jsonStr);
      }

so my keys look like this: "AP:201401:AZ5798BK"

When I try a lookup like:

    var keys = server.Keys(pattern: "AP:201401:*"); // returns in milliseconds
    var keyAry = keys.ToArray(); // returns in over one minute
    (note: this returns 12 keys)

it takes 1 min 12 seconds to return the keys. Once I have the keys it takes milliseconds to retrieve the values for those. If I iterate over the value of keys and return the values I get a similar result. I did a ToArray() just to isolate the problem.

If I try the same query in redis-cli.exe it comes back in milliseconds.

Am I using this command incorrectly?

Weej
  • 1,012
  • 2
  • 8
  • 24

3 Answers3

35

server.Keys automatically selects between KEYS and the preferred SCAN based on server version. I suspect what is happening is that you are thus using SCAN with a too-small page-size. There is an optional parameter for page size. Try specifying something considerably larger than the default - hundreds, thousands, etc. If not specified, the page-size uses the Redis SCAN default of 10, which could cause a lot of round-trips to be required.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • 1
    Thanks Marc I'll give that a try and let you know how it went. That might explain why the redis-cli.exe is so much quicker. – Weej Oct 19 '14 at 01:04
  • 3
    Marc! Yes that was the answer. A follow-up question: I did a little optimization testing and found that I continued to get performance boosts with pages sizes through 2000 though it leveled off around 1650. At that point it leveled off at around 180 milliseconds. Two questions: Is pagesize optimization a function of the db size or keysize? Second question: does memory allocation for the pagesize happen on the client or the server? Thanks very much for your answer! – Weej Oct 19 '14 at 11:06
  • This helped me solve a similar performance issue with `server.Keys`. Thanks! – G0dsquad Feb 07 '17 at 10:54
  • @weej the trick with setting larger page size dramatically improved performance in our solution! Thx! – sabiland Dec 31 '18 at 06:32
11

Do not use KEYS - it is a blocking command that will Render your Redis server unavailable to other request while it is running. Quoting from the command's documentation:

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using SCAN or sets.

If you read the warning thoroughly, you'll notice the recommended approaches at the end of the paragraph, namely using SCAN or Redis' sets. SCAN is non-blocking but since your question suggests that that you're into performance, I recommend using sets.

The idea is to maintain a Redis set with all the key names that are associated with that "pattern", so in your example you have to do the StackExchange.Redis-equivalent of SADD AP:201401 AP:201401:AZ5798BK after your call to cache.SetString, e.g.:

cache.SetAdd(wksYYMM, wksKey);

Disclaimer: I am not a C# programmer nor am I too familiar with StackExchange.Redis (sorry Marc ;))

Now, instead of KEYS or SCAN to get your keys, just do SMEMBERS AP:201401 or probably:

var keys = cache.Members(wksYYMM);

Bonus: since you're actually interested in the values of the these keys, you can use Redis' Lua scripting to fetch the keys values based on the set's members, or just use SORT.

Pure Redis:

SORT AP:201401 BY nosort GET *`

C# & StackExchange.Redis:

vals = cache.Sort(wksYYMM, by = "nosort", get = "*");
Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
  • 6
    server.Keys actually uses SCAN automatically if available. More info on separate answer. – Marc Gravell Oct 18 '14 at 13:15
  • I'm pretty happy with the implementation, I must admit. The enumerable sequence it hands you back scrolls / pages automatically as you work through the data. – Marc Gravell Oct 18 '14 at 14:49
  • Itamar, thanks for your response. I'm not sure sets will be appropriate because I actually want to update the items individually and add new ones as they're created. My understanding is that a set is a single storage item that holds multiple items. Eventually I want to use pub-sub to update those users interested in keys starting with AP:201410 when a new entry is made in the database. Then I'd send just that item via SignalR. For those reasons I want to keep them as separate items. – Weej Oct 19 '14 at 01:01
0

StackExchange.Redis server.Keys without set pageSize will be very slow.

Try this

server.keys( database:0, pattern:"error:*", pageSize: 5000);
DanielZ
  • 303
  • 3
  • 14