How to use message queueing in a real-life scenario?

Question

Ok, so I'm interested in message queueing. I really did loads of research on this subject already. I read 'programming windows azure' (about Azure Queues), I read loads of tutorials and information about the Azure service bus, I watched channel 9 videos on messaging patterns, etc.

But what I just don't understand: how would one use this in a real-life scenario? All the examples just put a single string or an object with some data in a queue and they read it from the queue on 'the other side'. But how do you know what to do with that data? For instance: I could try to save a Customer, an Order and an Address to the database, so I put these 3 objects in a queue to read the on the other side a put them in my database. How do I know what to do with these kind of objects?

Just some questions I have:

1. What kind of data would one put in a queue. A command maybe, that is executed when it's read, so everything in the queue has the same interface and the object itself figures out what to do?
2. Between what layers would one use queues? I was thinking about putting stuff in the queue in the service layer (after it has been validated) and reading it in the data access layer so I can put it in the database.
3. How many queues should one make? Just one queue between the layers you want to connect, or multiple queues between them (maybe for different purposes, although I can't think of any)?
4. This form of loose coupling allows us to queue requests so they can be processed later (for example: when we'd like to restart the database). That's cool but what if I want to READ data instead of writing it? Then the database should be online. And should I read data through a queue or could my service layer just pull the data from the data access layer and pass it to the presentation layer?

I think that were most of the questions that were buzzing around in my head. I hope anyone can clear this up for me.

Answer 1

In the simplest example, imagine you serialize to the queue the action you'd like performed, then the serialized version of the arguments:

Q: Delete: User 5

which is fine since your Worker Role could read that and act accordingly. Doesn't seem like a big bonus in this example, as queuing probably took as long as the deletion would, but what if you wanted to do BI on all your users? Your web page or application would never refresh while waiting for a synchronous request to do the processing. But if you pushed that request to the queue:

Q: RunSome2HourProcess: User *

Now you can return that you've queued the operation, and go on your merry way, while your server that's pulling from the queue would be able to process at its leisure.

What's more is this enables better scale scenarios. What if that last message was split up? So rather than doing all users, you did something like:

QRunSome2HourProcess: User A*-M*

QRunSome2HourProcess: User N*-Z*

Two azure worker roles would split the load and process your information in parallel.

There are also a number of scenarios around retrying, work flows, cross-role asynchronous communications, etc., but this should provide you some reasons why queuing can be good in the right scenarios.

So, in these examples: 1. we've put actions and identifiers/queries in a queued item. The worker role poling from the queue would know how to deal with it.

1. You could use a queue between your service layer and processing layers. you could use a queue between processing layers. you could use a queue within the same layer. The sky's the limit

2. I've typically made 1 queue per operation. So in the above example, i'd have the Run2HourProcess Queue, and just write the parameters to it. And create a new queue for any other process type. But they're super flexible, so it's up to the developer.

3. Unless it's a long running read (ie. the thumbnail generation example in the samples or the result of slow data processing) you'd be just as fast to go the database.

Answer 2

1. Well, you can put whatever you want in there. It's just a message, so whatever your layers can agree on will work. We put in messages that indicate when emails should be sent, when videos need to be processed, etc.
2. I wouldn't necessarily think about queuing between layers, so much as queueing between user-facing, realtime processes (eg, web servers) and background processing. Use queueing to do things that you don't want your web roles taking the time to do while a user is waiting.
3. You can make as many as you want, really. If the queue messages have a field that says what they mean then you could hypothetically put everything in one queue, though I wouldn't necessarily recommend that. We've got about half a dozen queues: one for sending notifications, one for video processing, one for report generation requests, etc.
4. I would definitely not use a queue as data storage and retrieval. It's just the wrong tool for that job, like trying to use TCP to store data.

Note that a single worker instance (ie, VM) can read as many queues as you would like it to. You just have to write the code that way. If you have a lot of queues that don't see much traffic, that's a reasonable way to keep costs down. Another alternative would be to combine several message types into a single queue. That way you only have one place to look for messages.

Alternatively, you can have many workers reading from the same queue, in order to spread the work out over them evenly.

Answer 3

In addition to what Rob mentions, message queuing is also used to handle bursts of traffic that can wait (in queue) until a free worker becomes available.

I believe the Azure Service Bus, queues and topics leverage message queuing under the covers to accomplish the same. I'm also seeing hints that the AppFabric hosted Workflow Foundation might do the same as well.