0

I have a requirement in which I have to do delayed scheduling in Redis. The delay can be in hours. As discussed in this answer one way of achieving this as below:-

  1. Add jobs to a sorted set
  2. Dispatcher thread which checks sorted sets every N seconds e.g - ZRANGEBYSCORE jobs -inf, <current unix timestamp>
  3. Whenever some jobs are available move it to main job list where main job workers are blpop'ing.
  4. Delete the jobs from sorted set.

Can some one suggest some ways how can I modify the above approach to handle the below case:-

I can't afford to lose message or give duplicate jobs to main worker thread. In the above approach if the dispatcher thread dies after reading the message from sorted set, writing to main job list but before deleting the message from sorted set. Then the message will be delivered twice to the main job list?

One way I can think of is the dispatcher thread will write to an intermediate list rather than to main list. One more thread will keep on checking the intermediate list in every N seconds. Whenever some job is available move it to the main list by doing BRPOPLPUSH.

Can some one suggest some other better and optimal way?

Community
  • 1
  • 1
tuk
  • 5,941
  • 14
  • 79
  • 162
  • Do you really want to imlement a home-made message queueing system if you've a lot of ones already done and working? RabbitMQ, hello? :D – Matías Fidemraizer Nov 02 '16 at 20:52
  • We are already having one. We need to add this support of delayed scheduling. Due to some non technical issues we cannot move to rabbitmq now. – tuk Nov 03 '16 at 05:23

1 Answers1

1

I don't try it but I think you can use a Lua script to transfer job from sorted set to main job list and delete it from the sorted set. According to the documentation on transaction :

A Redis script is transactional by definition, so everything you can do with a Redis transaction, you can also do with a script, and usually the script will be both simpler and faster

And

All the commands in a transaction are serialized and executed sequentially. It can never happen that a request issued by another client is served in the middle of the execution of a Redis transaction. This guarantees that the commands are executed as a single isolated operation. Either all of the commands or none are processed, so a Redis transaction is also atomic.

LFI
  • 654
  • 5
  • 11
  • Lua script will block the entire redis when it is executing or it will block only the data structure on which it is operating? I am trying to ask is let's say in lua script I execute a command as `redis.call("zadd", 'test1', i, i)` so will this block the entire redis or it will block only the operations on `test1`? – tuk Nov 03 '16 at 09:11
  • I think everything is blocked until script completion, see the [doc](http://redis.io/commands/EVAL#atomicity-of-scripts) . I don't know if it's a problem, as what you are trying to do is simple. – LFI Nov 03 '16 at 09:52
  • If it blocks everything then shouldn't the approach I proposed in my question is a better one? – tuk Nov 03 '16 at 10:36
  • To complete me previous answer, every other commands (not I/O for example) are blocked until the lua script ends. As Redis is single threaded, this is the cost of the atomic operation you're trying to do. See this [post about it](http://stackoverflow.com/questions/10489298/redis-is-single-threaded-then-how-does-it-do-concurrent-i-o). – LFI Nov 03 '16 at 12:19