2

I recently created an error manager to take logged errors from clients on our network and put them into an MSMQ for processing. I have a separate Windows Service running on the server to pick items off the queue and push them into a database.

When I wrote it and tested it everything worked great; however I neglected to consider that at deploy-time, having 100 clients all sending to a public queue might not be performant, best-case, and worst-case there could be all kinds of collisions, it seems to me.

My thought right now is to front the MSMQ with a WCF service and make everyone go through that. The logic being that at that point I could employ some locking, etc. If I went with a service I think I could employ a private queue instead of a public one, which would be tons faster, as well.

What I'm not sure is, am I overthinking it? MSMQ is pretty robust and the methods I think are thread-safe. Should I just leave it alone and see what happens? If I do put in the service, how much management would I need to have in place?

Mike K
  • 1,313
  • 2
  • 18
  • 28

2 Answers2

1

I recently created an error manager to take logged errors from clients on our network and put them into an MSMQ for processing

I assume you're using System.Messaging for this? If so there is nothing at all wrong with your approach.

having 100 clients all sending to a public queue might not be performant

MSMQ was designed from the bottom up to handle high load. Depending on the size of the individual messages and the storage threshold of the machine, a queue can hold 10's of thousand of messages without any noticeable performance impact.

Because a "send" in MSMQ involves the queue manager on each machine writing messages locally before transmission (in a store and forward messaging pattern), there is almost no chance of "collisions" or any other forms of contention happening; if the sender is unable to transmit the message it simply "sends" it to a temporary local queue and then the actual transmission happens in the background and is mediated by the fault tolerant and very reliable msmq protocol.

My thought right now is to front the MSMQ with a WCF service and make everyone go through that

This would be a valid choice if you were starting from nothing. As another poster has stated, WCF does hide you from some of the msmq-voodoo by removing the necessity to use System.Messaging. However, you've already written the code so I see little benefit exposing a netMsmqBinding endpoint.

If I went with a service I think I could employ a private queue instead of a public one

As far as I understand it from your description, there's nothing to stop you using a private queue in your current scenario. In fact I'd recommend always using private queues as they're much simpler.

If I do put in the service, how much management would I need to have in place?

You will have more management overhead with a wcf service. Because you're wrapping each end of a send-receive with the WCF stack, there is more code to spin up and therefore potentially fail. WCF stack exceptions are famously difficult to troubleshoot without full service logging enabled.

EDIT - in response to comments

I think for a private queue you have to actually be writing FROM the machine the queue sits on, which would not work in a networked environment

Untrue. MSMQ supports transactional reads to and writes from any private queue, regardless of whether the queue is local or remote.

This is because any time a message is sent from one machine to another in msmq, regardless of the queue address, the following happens:

  1. Queue manager on sending machine writes the message to a temporary local "outbound" queue.
  2. Queue manager on sending machine contacts queue manager on receiving machine and transmits the message.
  3. Queue manager on receiving machine places the message into the destination queue.

If you are using transactions, the above steps will comprise 3 distinct transactions.

Something to remember: the safest paradigm in exchanging messages between queues on different machines is send remote, read local.

So this means when you send a message, you're instructing msmq to send to a remote queue address. However, when someone sends something to you, they must do the same. So you end up reading only from local queues, and sending only to remote queues.

This way you get the most reliable messaging setup, because when reading, a local queue will always be available.

Try it! I've been using msmq for cross machine communication for nearly 10 years and I've never used a public queue. I don't even know what they're for!

tom redfern
  • 30,562
  • 14
  • 91
  • 126
  • Wow, great comments. So it sounds like you're saying the MSMQ all by itself is sufficient to deal with multiple incoming Sends. So I can leave that alone. As to the private/public queue, I think for a private queue you have to actually be writing FROM the machine the queue sits on, which would not work in a networked environment. Unless I'm missing something. In my tests, a private queue was a thousand times faster than a public one. – Mike K Jan 14 '15 at 23:03
0

I would expose an WCF "IsOneWay" method. And then host your WCF in IIS.

The IsOneWay will wire up to MSMQ.

This way...you have the robustness of IIS hosting. You can expose any endpoint you want. But eventually the request makes it to MSMQ.

One of hte reasons is the ease of using msmq with wcf. Having written and used msmq "pre-wcf" I found the code (pulling messages off the queue and error handling) to be difficult and problematic. That alone would push me to WCF hosting.

And as you mention, the security around a local-queue is much easier to deal with.

Bottom line, let WCF handle the msmq-voodoo for you.

Simple example below.

[ServiceContract]
public interface IMyControllerController
{

    [OperationContract(IsOneWay = true)]
    void SubmitRequest( MyObject obj );

}

http://msdn.microsoft.com/en-us/library/ms733035%28v=vs.110%29.aspx

http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontractattribute.isoneway%28v=vs.110%29.aspx

What happens in WCF to methods with IsOneWay=true at application termination

http://blogs.msdn.com/b/tomholl/archive/2008/07/12/msmq-wcf-and-iis-getting-them-to-play-nice-part-1.aspx

Community
  • 1
  • 1
granadaCoder
  • 26,328
  • 10
  • 113
  • 146
  • I like this attribute for use on a service like this, that makes sense. I'm not sure what you mean by MSMQ working more easily inside IIS. I've already written the code that works with the queue, it was pretty straightforward. My main concern is contention. Say I have 50 clients all wanting to hit the queue at the same time. 1) Am I better off with them going to a service instead of directly to a queue? 2) Inside the guts of the service, do I need to implement any locking to gatekeep this one resource as I try to put 50 calls into it? Is IIS governing the incoming calls at all? – Mike K Jan 13 '15 at 22:40
  • IIS. Like, if you host it in IIS, then you get the auto-start-up with a computer reboot. It is easier to horizontally scale. Take the msmq part out of it...I mean...."I prefer hosting WCF apps in IIS"...that's the take away there. – granadaCoder Jan 13 '15 at 22:45
  • If you have WCF/MSMQ.....with 50 clients......that really sounds like pocket change. Remember, the request only has to queue to the message. Not process it. I did a test once, but I was able to throw thousands of messages at it, IIRC. Keep in mind, this was 2006 when I did my first wcf app with msmq. (I've done many wcf apps since then). – granadaCoder Jan 13 '15 at 22:47
  • the netMsmqBinding has some options for controlling throttling : http://msdn.microsoft.com/en-us/library/ms731380%28v=vs.110%29.aspx – granadaCoder Jan 13 '15 at 22:48
  • 1
    I added one more link (bottom of my post) – granadaCoder Jan 13 '15 at 22:49
  • The system I worked on had 1,000 concurrent users.....submitting asych requests about every 1-2 minutes in a scattered fashion through wcf. In my tests, I wired up a test-app to spawn N number of threads and just submit requests to try and kill the my service.....it held up. I don't remember the exact number of requests/threads per second, but IIRC, it handled it fine. – granadaCoder Jan 13 '15 at 22:53
  • *The IsOneWay will wire up to MSMQ.* - what does this statement mean? There is nothing at all intrinsically "msmq" about a one way service operation. – tom redfern Jan 13 '15 at 22:56
  • If the OP uses IIS as a host then what happens when the app pool unloads and someone sends a message onto the queue? There won't be any in-process handler listening for the message. There is no benefit to using IIS over a windows service in the OP's scenario. – tom redfern Jan 13 '15 at 23:08
  • *the security around a local-queue is much easier to deal with* - I assume by "local queue" you mean "private queue"? – tom redfern Jan 13 '15 at 23:22
  • Yes, I meant "private-queue". "Wire up to MSMQ". What I mean is that WCF will handle the writing the messages to the queue, and reading the messages to the queue. you don't have to write the plumbing. The wcf will create the private-queue if it doesn't exist and will write messages and read messages. And with the netMsmqBinding, you have options. IIS vs Windows-Service, to each his own. – granadaCoder Jan 14 '15 at 14:14
  • I'm sorry to say this, but what are you talking about? jquery sending a msmq message? You do know that jquery runs in a browser right? How can a browser access the msmq subsystem?? Your answer makes no sense, and your comments even less. – tom redfern Jan 15 '15 at 09:16
  • When you expose a WCF service over netMsmqBinding, by definition you need a msmq client to call them. The *transport* that you expose the service over *is* MSMQ. If you don't understand this then you have no business answering a question about msmq. – tom redfern Jan 15 '15 at 09:42
  • You are correct. I made a mistake. " NetMsmqBinding: Sends binary-encoded SOAP over MSMQ. This binding can only be used for WCF-to-WCF communication." – granadaCoder Jan 15 '15 at 14:28