1

I'm looking into redis for my specific need but I know don't see how I could achieve what I need.

I want to store sets of integers (100s-low thousands entries per set) and then "query" by an input set. All sets that contain all the elements as the query should match. (SDIFF query key should return empty set and then iterate over all keys).

I don't see how this can be done efficiently (about 5ms per 10k keys)?

beginner_
  • 7,230
  • 18
  • 70
  • 127

1 Answers1

2

If you will only query your data by integer, consider then storing using the integer as a key. Instead of:

SADD a 3 5
SADD b 3 7

You can:

SADD int:3 a
SADD int:5 a
SADD int:3 b
SADD int:7 b

Then you use SINTER int:3 int:7 ... to get all matching integer set names (what you used for keys originally).

If you do need to query both ways, then you may need to do both. This is like doing a many-to-many relationship in Redis.

Here, you are trading off insert time and memory usage for fast query performance. Every time you add a pair, you need to both SADDs: SADD setName int and SADD prefix:int setName.

If this extra memory and insert effort is not an option on your case, your next option is to use a Lua Script to loop through the keys (pattern matching your set names) and using SISMEMBER to test through the integers of your query. See Lua script for Redis which sums the values of keys for an example of looping through a set of keys using Lua.

A Lua script is like a stored procedure, it will run atomically on your Redis server. However, if it will give perform at 5ms for 10k sets tested for multiple integer members remains to be seen.

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
  • This works as intended and hence marked as accepted answer. The performance however is far from 10k per 5ms (which kind of is to be expected as network is involved, it's more like 1 second per 10k). For the test set it's twice as fast than doing a similar thing in elastic search. However given the description of the time complexity of SINTER (O(N*M)) I'm rather convinced this will not scale. – beginner_ Jan 07 '20 at 06:22