3

We have a number of MSMQ queues throughout our system, both private and public queues. Sometimes a windows service that reads from a queue will crash, and so messages will build up in that queue. Once the queue gets to a certain size (maybe 60K messages), all queues on that server will stop working, throwing errors about insufficient resources.

My question is, how are the queues really working behind the scenes, are they storing messages in RAM or on the hard drive? Does it run out of resources and crash when the server runs out of RAM? If it's using some allocated space on the hard drive, is there a way to increase the allowable size? If it's using RAM, can I simply add RAM to the servers and then that will increase the allowable size?

I need to make sure that when a service goes down, we can handle storing 100K or 200K messages in that queue while we work on fixing the service, as those messages are critical to our business.

Justin
  • 17,670
  • 38
  • 132
  • 201
  • 60,000 messages is a trivial amount so your server must already be very low on kernel memory before you even begin. Adding RAM won't help as most modern machines have enough RAM fitted to maximise kernel memory. You need to instead identify and remove the kernel memory hogs. A well configured machine can handle millions of messages. – John Breakwell Nov 09 '11 at 22:45

2 Answers2

2

Here is an article on MSDN that seems to address your question (as John points out below, this only applies to Windows Server 2000 so should probably be ignored by most people): Resource management in MSMQ applications. Specifically:

For MSMQ 1.0 and MSMQ 2.0, the combined size of messages capable of being stored on one machine is not limited to the amount of RAM in the machine or the size of the hard disk, but to the amount of virtual address space provided to the MSMQ service by the operating system (this limitation has been lifted in MSMQ 3.0). Each process in an x86 machine is allotted a virtual 4 GB of addressable memory. 2GB is reserved for use in kernel mode and 2GB for user mode. The MSMQ Queue Manager operates in user mode and therefore has an addressable 2GB of virtual address space to work with. Each message's data is stored in RAM, which is backed up by the system's paging file or memory mapped files. MSMQ uses memory mapped files to store both express and recoverable messages. Since we are limited to 2GB of addressable memory, we are limited to 2GB worth of messages on a disk. When you take into account the memory utilized by MSMQ code and its internal data structures, as well as file allocation to store message files on disk, we end up with between 1.4GB and 1.6GB worth of messages that can be stored on disk.

Note   This limitation of 1.6GB can be raised to approximately 2.6GB by enabling 3GB tuning on the MSMQ Service. See Q171793 for more information on how to enable 3GB tuning.

Edit: the tuning link seems to be broken. I believe it should be pointing here.

In terms of later versions of MSMQ, John discusses the issue in a blog post.

Maximum number of messages

This one is not as simple to work out. From my Insufficient Resources post we know that each message needs 75 bytes of kernel memory for indexing so, for example, 2 million messsages would require roughly 150 megabytes. It would seem, therefore, that all you need to do is add more RAM. After looking at a comparison of 32-bit and 64-bit memory architectures, though, you will quickly have to move to the 64-bit platform to take advantage of your investment as 32-bit machines max out at 450 MB of paged pool memory regardless of the amount of RAM fitted.

But, again, if you are trying to work out what amount of RAM will generate the paged pool memory required to accommodate a billion MSMQ messages, your design spec is up for some serious reviewing.

Community
  • 1
  • 1
Rohland
  • 1,405
  • 14
  • 20
  • The link on tuning does seem to be broken. It's directly referenced from the article unfortunately :/ I did find another helpful article with far more detail on the matter: http://blogs.msdn.com/b/johnbreakwell/archive/2006/09/18/insufficient-resources-run-away-run-away.aspx – Rohland Nov 09 '11 at 21:23
  • 2
    Better than that blog article is ""What are MSMQ's limits?" If I had a farthing for every time..." http://blogs.msdn.com/b/johnbreakwell/archive/2008/02/29/what-are-msmq-s-limits-if-i-had-a-farthing-for-every-time.aspx – John Breakwell Nov 09 '11 at 22:37
  • The MSDN paragraph originally quoted above should be ignored as it only applies to Windows 2000 (MSMQ 2.0) and before. If I could, I would have the article burnt as it now just misleads instead of inform. – John Breakwell Nov 09 '11 at 22:41
0

Not sure about the in-depth answer, but on a surface level anyhow, a non-transactional queue stores messages in memory, whereas a transactional queue stores messages on disk.

UPDATE

As John states below, all messages are held on disk whether durable or non-durable queues are used.

tom redfern
  • 30,562
  • 14
  • 91
  • 126
  • 1
    That is incorrect. All messages are written to disk. Transactional and Recoverable messages are written to disk in such a way that they can be recovered after a reboot. Express messages are lost after a reboot. http://blogs.msdn.com/b/johnbreakwell/archive/2008/04/04/why-are-my-express-messages-being-written-to-disk.aspx – John Breakwell Nov 09 '11 at 22:35
  • 1
    Thanks John. Would seem to be non-intuitive but I certainly take your word for it with your background in the product. – tom redfern Nov 10 '11 at 08:13
  • 1
    Yes, indeed. I'd been telling people the wrong thing for years before one of the original architects corrected me. – John Breakwell Nov 10 '11 at 10:26