0

I have been trying to get all the keys from Redis using scan while also avoiding using count. Problem is, when there are many keys (ex. 1000000) scans SCAN 0 MATCH * are much slower than:

KEYS *

or

SCAN 0 MATCH * COUNT 1000000

I have read an article saying to use Lua for better performance. Can anybody help me regarding this matter? I want to use Lua to handle a scan of all the keys where the number of keys is more than 1M. I am using the redis package by npm.

lazysnoozer
  • 85
  • 11
  • I don't get it, you says you want to use scan but it way more slower than *SCAN 0 MATCH * COUNT 10000000 *, these are not the same thing?? – Sándor Bakos Feb 12 '20 at 08:03
  • It's the same thing but when the users will input the data and keys, then I don't know how many keys are there. So, then I don't know what to set in the count. I mean if put the less value in the count than the keys in the db then its all the same, right ? – lazysnoozer Feb 12 '20 at 08:56

2 Answers2

0

Using SCAN with COUNT 1000000 would be slightly worst than using KEYS.

Redis is single-threaded. One of the reasons SCAN was introduced is to allow going through all the keys without blocking the server for a long time, by going a few steps at a time.

The default value is 10. It means the command will bring back more or less 10 keys, could be less if the keys are sparsely populated in the hash slots, or filtered out by the MATCH pattern. It could be more if some keys are sharing a hash slot. Anyhow, the work performed is proportional to the COUNT parameter.

This doesn't mean the KEYS command (or a SCAN with a big COUNT number) is forbidden. If you are ok with your server being blocked for the time it takes to run, go ahead.

See Is there any recommended value of COUNT for SCAN / HSCAN command in REDIS? for more details and a Lua example.

In It is possible to merge two separate DBs (e.g. db0, db1) running in one single Redis instance (6379) into one single DB (db0)? and How can I get all of the sets in redis? I have posted examples of how to use SCAN within a Lua script to achieve custom requirements.

You may find Redic `SCAN`: how to maintain a balance between newcomming keys that might match and ensure eventual result in a reasonable time? interesting.

How to use Lua in Redis with node.js: executing redis eval command to run Lua script in nodeJS

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
  • Thanks for clearing out the confusion regarding count. I have used scan without using count and its way too slow. That's why switched to Lua for better speed. But still it is slower than `KEY` . Is `Lua` recommended for more than one action? Like getting the keys as well as data? And is `Hash` more preferable with `Lua` for better performance ? – lazysnoozer Feb 13 '20 at 06:03
  • Lua is recommended for more than one action, but you can also use MULTI/EXEC. If a `Hash`is better depends totally on the requirement, would need more info. – LeoMurillo Feb 14 '20 at 16:07
-2

Found related solution. Please go through this link. Using of Keys is not advised and also if you go through the documentation you can see that. Though here is the performance is given but it's for the sake of understanding more deep knowledge, nothing else. The best way is to use Lua Script which is advisable and also the performance is really good. And if you see the comments you will also get to know about the difference as well, I guess. The performance for the KEY, SCAN , Lua Script :

  • KEYS: the fastest method
  • SCAN: ~1.5 times slower
  • SCAN with lua:~2.4 times slower
  • SCAN with lua and additional checks: ~4.3 times slower
  • KEYS with lua: ~18 times slower

For me, it was quite difficult to understand the syntax of Lua. So, here is the code to help understanding the syntax.

Scan without count in Lua Script: 

local ans, has, cursor = {}, {}, "0";
repeat
    local t = redis.call("SCAN", cursor, "MATCH", KEYS[1], "COUNT", 1000000000);
    local list = t[2];
    for i = 1, #list do
        local s = list[i];
        if has[s] == nil then has[s] = 1; ans[#ans + 1] = s; end;
    end;
    cursor = t[1];
until cursor == "0";
return #ans; --or return ans;
lazysnoozer
  • 85
  • 11