5

For my ongoing project, I am using Redis for message distribution across several processes. Now, I am supposed to make them reliable.

I consider using the Reliable queue pattern through BRPOPLPUSH command. This pattern suggests that the processing thread remove the extra copy of message from "processing list" via lrem command, after the job has been successfully completed.

As I am using multiple threads to pop, the extra copies of popped item go into a processing list from several threads. That is to say, the processing queue contains elements popped by several threads. As a consequence, if a thread completes its job, it cannot know which item to remove from the "processing queue".

To overcome this problem, I am thinking that I should maintain multiple processing queues (one for each thread) based on threadId. So, my BRPOPLPUSH will be:

BRPOPLPUSH <primary-queue> <thread-specific-processing-queue>

Then for cleaning up timedout objects, my monitoring thread will have to monitor all these thread specific processing queues.

Are there any better approaches to this problem, than the one conceived above?

Mopparthy Ravindranath
  • 3,014
  • 6
  • 41
  • 78
  • IMO that makes sense. If I was that worried about reliability however replacing Redis for this action would make sense. – Joe Doherty Apr 18 '15 at 08:43
  • I was thinking about the same problem and I ended up with the same solution, but where to store the exact last modification time of each "thread-specific-processing-queue" in order to understand which one should be cleared? – MaxChinni Jul 27 '15 at 11:34
  • I ended up storing in which is a redis z-set, and not a queue. So the expiry time is the score of the element in the z-set. Then I use redis scheduling pattern to scan the z-set based on score and then put back the expired keys to main queue. – Mopparthy Ravindranath Jul 28 '15 at 08:20
  • @MupparthyRavindranath I have a similar requirement, what do you mean by 'redis scheduling pattern'? – user779159 Nov 05 '16 at 14:55
  • @MupparthyRavindranath : so are we to say that the BRPOPLPUSH implementation people never thought about the multithreaded aspect of processing such messages because i am facing the same issue when thinking about it in a C# client implementation. – ankur Nov 19 '17 at 12:29

1 Answers1

3

@user779159

To support reliable queue mechanism, we take the following approach:

 - two data structures
    -- Redis List (the original queue from which items are popped regularly)
    -- a Redis z-set, which temporarily stores the popped item.

Algorithm:

-- When an element is popped, we store in z-set 
-- If the task that picked the item completed its job, it will delete the entry from z-set.
-- If the task couldn't complete it, the item will be hanging around in z-set. So we know, whether a task was done within expected time or not.
-- Now, another background process periodically scans this z-set, picks up items which are timedout, and then puts them back to queue

How it is done:

  • we use zset to store the item that we poped (typically using a lua script).
  • We store a timeout value as the rank/score of this item.
  • Another scanner process, will periodically (say every minute) run z-set command zrangebyscore, to select items between (now and last 1 minute).
  • If there are items found by the above command, this means the process that popped the item (via brpop) has not completed its task in time.
  • So, this 2nd process will put the item back to the queue (redis list) where it originally belonged.
Mopparthy Ravindranath
  • 3,014
  • 6
  • 41
  • 78