5

I am currently in the process of putting together a reference architecture for a distributed event-based system where events are stored in a SQL Server Azure database using plain old tables (no SQL Server Service Broker).

Events will be processed using Worker Roles that will poll the queue for new event messages.

In my research, I see a number of solutions that allow for multiple processors to process messages off of the queue. The problem I have with a lot of the patterns I'm seeing is the added complexity of managing locking, etc when multiple processes are trying to access the single message queue.

I understand that the traditional queue pattern is to have multiple processors pulling from a single queue. However, assuming that event messages can be processed in any order, is there any reason not to just create a one-to-one relationship between a queue and its queue processor and just load-balance between the different queues?

queue_1 => processor_1
queue_2 => processor_2

This implementation avoids all of the plumbing necessary to manage concurrent access to the queue across multiple processors. The event publisher can use any load-balancing algorithm to decide which queue to publish messages to.

The fact that I don't see this sort of implementation in any of my searches makes me think I'm overlooking a major deficit in this design.

Edit

This post has triggered a debate over using database tables as queues vs. MSMQ, Azure Queues, etc. I understand that there are a number of native queuing options available to me, including Durable Message Buffers in Azure AppFabric. I've evaluated my options and determined that SQL Azure tables will be sufficient. The intention of my question was to discuss the use of multiple processors against a single queue vs. one processor per queue.

David Makogon
  • 69,407
  • 21
  • 141
  • 189
  • "So, what am I missing?" I would suggest that you're missing the whole point of using a proper event queue. Building an event queue using a database seems silly when event queues are already first-class products. Why not use MS-MQ and save yourself a lot of pain? – S.Lott May 04 '11 at 18:47
  • @S. Lott: I would argue that there is always a reason to have your queues and data in the same store. Uniform backup/restore, elimination of two-phase commit with every operation (DTC between message store and data store), one product to deploy/troubleshoot/administer, one HA/DR solution which fails over the message store and data store together in a consistent state, all these and more make a very compelling case for queues inside the database. Considering that almost every message starts as a result of a data operations and ends up updating data, events *are* data and they belong together. – Remus Rusanu May 04 '11 at 19:43
  • @S.Lott: 1) I don't have MSMQ, as I'm deploying on Azure. 2) MS-MQ has plenty of its own pain. –  May 04 '11 at 20:26
  • @tferreira: You asked what you were missing. It appears your missing Azure Queues: http://msdn.microsoft.com/en-us/library/dd179363.aspx. Please **update** the question to explain why you aren't using the obvious solution. – S.Lott May 04 '11 at 20:30
  • @S.Lott: Because it's not obvious. Azure Queues have a number of limitations, not the least of which is transactional integrity. How many solutions have you rolled on Azure Queues? How many using database queues. I'm looking for people who have real experience doing this. Thanks. –  May 04 '11 at 20:52
  • @tferreira: All of the database queues I've seen have been terrible, terrible messes. Really bad. Why do you say Azure Queues lack transactional integrity? Can you **update** the question with this additional information so that it's clear why you're not using them? – S.Lott May 04 '11 at 20:55
  • You've mentioned a few times in this question that Azure Queues aren't transactional. I curious as to what you mean by that. – knightpfhor May 05 '11 at 01:23
  • 1
    MSMQ offers transactional queues. Azure queues use a timeout mechanism as descibed here http://cloudshaper.wordpress.com/2010/11/05/understanding-the-transactional-behavior-of-an-azure-message-queue/ –  May 05 '11 at 03:11
  • Discussion of Azure Queue limitations http://abdullin.com/journal/2010/4/6/nservicebus-and-windows-azure-challenges.html –  May 05 '11 at 13:25

4 Answers4

5

See Using tables as Queues for a more detailed discussion of this topic. The issue is not only how you access the 'queue', but also how you index it, the clustered index must allows direct seek of the next row to dequeue, otherwise you'll deadlock constantly.

You want your processors to race to the same queue, load balancing by spreading out to different queues is an anti-pattern. It leads to convoys and artificial latency where you have items queued up behind a late processor, but other processors are free and idle because their queue is empty.

Remus Rusanu
  • 288,378
  • 40
  • 442
  • 569
  • I'm not sure load balancing by having multiple queues would constitute an anti-pattern, as I'm merely horizontally partitioning the table, which is common. WRT convoys and late processors, I'm assuming a proper load-balancing algorithm would distribute publications to each queue in a way that would keep them balanced. –  May 04 '11 at 20:34
  • If by partitioning you imply scale-out of the message store then multiple queues is the only way. But if you plan to deploy multiple queues in one store (ie. one Azure DB) then I stick to my opinion than one queue is better than multiple. The scalability of a single queue, when properly done, is far greater than a single Azure DB can drive, so just isn't any reason to have more queues. – Remus Rusanu May 04 '11 at 21:31
  • Good input. I will take your arguments into consideration. Thanks for the feedback. –  May 04 '11 at 21:48
1

Tables as queues are quite easy to do. See my SO answer here please: SQL Server Process Queue Race Condition

Community
  • 1
  • 1
gbn
  • 422,506
  • 82
  • 585
  • 676
1

As S.Lott mentioned, there are message queue mechanisms you can use. MSMQ won't really help in Windows Azure, but Windows Azure already has a durable queue mechanism. You can easily set up each worker role instance to read one (or more) queue items. Once a queue item is read, it's "invisible" for whatever length of time you specify (or 30 seconds if no time specified). Queue messages can be up to 8K, and they're considered "durable" - all Azure storage is replicated a minimum of 3 times (as is SQL Azure).

While you can implement something like what gbn describes, I really think you should consider the native Azure Queue service when working in Windows Azure. You'll easily be able to scale to multiple queue consumers and won't have to worry about concurrency or special load-balancing code - just increase (or decrease) instance count.

For more info about Windows Azure queues, check out the Azure Platform Training Kit - there are several simple labs that walk you through queue basics.

David Makogon
  • 69,407
  • 21
  • 141
  • 189
  • Azure Queues are an option I'm looking into. However, I'm concerned with the fact that they are not transactional. –  May 04 '11 at 20:39
  • From an "understanding the core underpinnings of Azure Queues" - yes, absolutely a useful link. Keep this in mind: There's a complete, official .NET SDK that hides all of the REST API so you don't have to worry about that (still useful to understand though). There's also libraries for php and Java, and a few open source projects for Ruby and python as well. – David Makogon May 04 '11 at 20:40
0

The point you're missing, to my mind, is that when using queues one of the important point is that orders are saved and whatever happens once it's in the queue it won't be lost.

Now pollers process can die, they wan have a lot of different problems, you don't care, the queue is the place where the orders are safe.

Pollers does'nt require the same level of robustness. Postfix for example is a very secure implementation of mail transporter where message queues are used in a lot of levels (each subsystem in the application which requires a different security level communicate with others with queues) - and you can switch off the power you will not loose any mail, workers can die very badly, mails can't.

Edit

That means the basic usage is storing an order, and ignoring what the workers will do with that, how many workers are still alive, etc. So the only reason to handle several queues is to manage several destinations for your order (application logic) and not to manage the way the workers should work with them (Decoupling).

regilero
  • 29,806
  • 6
  • 60
  • 99