2

We have some Redis keys with a given TTL that we would like to subscribe to and take action upon once the TTL expires (a la job scheduler).

This works well in a single-host environment, and when you subscribe in ServiceStack, using its Redis client, to '__keyspace@0__:expired', that service will pick it and take action. That's fantastic...

... until you have a high-availability topology set up, with more than one API instance in that cluster. Then every single host appears to be picking up on that message and potentially doing things with it.

I know keyspace notifications don't work exactly the same as traditional pub/sub or messaging-layer events, but is there a way to perform some kind of acknowledgement on these kinds of events, so that, at the end of the day, only one host will carry on with the task?

Otherwise, is there a way to delay a message publishing?

Thanks!

napo
  • 869
  • 9
  • 19

2 Answers2

2

As describe in https://redis.io/topics/notifications

very node of a Redis cluster generates events about its own subset of the keyspace as described above. However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. Put differently, keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.

So client should create separate connection to each node to get redis keyspace notification.

Hlex
  • 833
  • 8
  • 16
1

My understanding of your question: You need an event based unicast notification whenever a key is expired.

This solution will be helpful to you if above assumption is correct. It's kind of crude solution but works!

Solution: You need to put(may be using a service/thread) the expired keys in the Redis List/queue. Then blocking B*POP operation from the client instances on this list/queue will give you what you want!

How does it work? Let's assume, a single background thread will continuously push the expired keys into a redis list/queue. The cluster of API instances will be calling blocking pop on this list/queue.

Since, blocking pop operation on each item of redis list will be consumed by only one client, only one API instance will the get the notification of expired key!!!

Ref:

List pop operation: https://redis.io/commands/lpop

Similar problem with pub/sub: Competing Consumer on Redis Pub/Sub supported?

code
  • 2,283
  • 2
  • 19
  • 27
  • Thanks for noting it. Hope I have clarified it. For instance, there could be a system that puts the key on redis in the first place. That system could move the key to a list upon expiry. – code Jun 11 '20 at 15:16
  • 1
    I think this is the real struggle, how can I define such functionality that upon key expiration -> move key into list/queue. – Etian Chamay Jun 11 '20 at 19:43
  • @EtianChamay - Indeed, you would have a service that receives the key-event notification and then pushes the expired key to a list (`RPUSH`), then the consumer can use `BLPOP` to consume that data. When in an hight availability environment, this service will be duplicated, thus `RPUSH` will be triggered twice for the same key... – Hollyol May 10 '22 at 08:56