How do I store a large number objects in RedisJSON?

Question

Say I have a JavaScript object structured like so:

{
  id: "someUniqueId",
  foo: "otherData",
  bar: 30
}

I want to store hundreds of these in RedisJSON. I am currently initializing a key 'json-objects' as an empty object using the node-redis client:

await client.json.set('json-objects', `$`, {});

I am then adding objects to that key's root object like:

const dataToStore = {
  id: "someUniqueId",
  foo: "otherData",
  bar: 30
}

await client.json.set('json-objects', `$.${dataToStore.id}`, dataToStore as any);

This results in a big object under they key 'json-objects' that looks like:

{
  "someUniqueId-1": {
    id: "someUniqueId-1",
    foo: "otherData",
    bar: 30
  },
  "someUniqueId-2": {
    id: "someUniqueId-2",
    foo: "otherData2",
    bar: 31
  },
  "someUniqueId-3": {
    id: "someUniqueId-3",
    foo: "otherData3",
    bar: 32
  },
  and so on...
}

Which I can then query and update using paths:

await client.json.set('json-objects', `$.${id}.foo`, "someUpdatedData");

My question is: Is this the best way to store the data, or would it be more efficient to store each object at it's own RedisJSON key, or in another way? My only issue with doing it this way is how do I differentiate objectTypeA from objectTypeB when I inevitably need to cache other data types in RedisJSON? Could I prefix the key with a data type? Something like below:

const dataToStore = {
      id: "someUniqueId",
      foo: "otherData",
      bar: 30
    }

await client.json.set(dataToStore.id, `$`, dataToStore as any);
// -OR-
await client.json.set(`objectTypeA-${dataToStore.id}`, `$`, dataToStore as any);

This just doesn't feel like the preferred way to do this. Maybe what I'm doing now is actually not that inefficient with JSON path lookups?

I suppose somthing like multiple Redis instances for different collections could be a solution. Though this seems like a lot of extra overhead. https://stackoverflow.com/questions/16221563/whats-the-point-of-multiple-redis-databases — bapin93, Aug 26 '22 at 15:30

score 6 · Accepted Answer · answered Aug 26 '22 at 16:03

Spreading it out over multiple keys is the way to go here.

Having everything in one big key means that that one big key is on a single shard. Not a huge problem when you just have a single node. But when it comes time to scale and you need a cluster, that key will still be on a single shard. The other shards in the cluster will not be used. And thus it won't scale.

This is what we call a "hot key" and can be the source of all sorts of pain and trouble.

The idea of prefixing the types in the key name is a common pattern in Redis and is a very suitable way to handle this. We call them keyspaces. So, solid idea.

Typically, we use colons to delimit the keyspaces. So, if I were storing say Bigfoot sightings and UFO sightings and Redis, you might use keyspaces like:

sightings:bigfoot:12345
sightings:ufo:67890

The colons are just a convention—use whatever you like. Don't let me tell you how to live you life!

Thank you, I realize moving to this approach will also allow me to better utilize keyspace notifications for specific objects, as well as the prefixing allowing for keyspace notifications for entire or parts prefixes using pattern matching. — bapin93, Aug 30 '22 at 21:00
It also enables the usage of RediSearch to create indices to find specific objects. — Guy Royse, Sep 01 '22 at 13:47

How do I store a large number objects in RedisJSON?

1 Answers1