28

I have a windows service that is currently instantiating about a dozen FileSystemWatcher instances to monitor shared folders across the corporate network for files to be processed.

I am looking into adding more instances so I'm wondering if anyone here has experience (with production systems) as to what are practical limits on number of FileSystemWatcher instances a production system can reliably handle?

Edit: In my case, the InternalBufferSize property is not modified so the InternalBufferSize is the default 8 KB... I assume the increase in InternalBufferSize would affect the number of FileSystemWatcher instances a system can run simultanesouly so that is also a part of the equasion...

Edit: If you think that this is exclusively a resource issue and it only depends on the amount of available memory or some other hardware aspect of the system, please share your experience or links to documentation or articles that corroborate your opinion... I would really like to hear from someone who reached the limit in production regardless of their hardware specs so please before voting to close consider that 7 other people in less than 20 minutes have shown interest in hearing from someone who pushed the limits on this...

Dean Kuga
  • 11,878
  • 8
  • 54
  • 108
  • Why not use one or few and just filter out inside - like suggested in the documentation `To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications` - would that work? – NSGaga-mostly-inactive Apr 17 '12 at 16:42
  • @NSGaga, the watched folders are on different servers accross the corporate network... – Dean Kuga Apr 17 '12 at 16:44
  • you should add that I think - might help your question as well (not sure why myself, looks ok to me) as it makes it a more unique problem. So you have e.g. 'one per remote machine'? – NSGaga-mostly-inactive Apr 17 '12 at 16:46
  • @NSGaga, it is one or more per remote machine depending on the shared folder structure... question modified. – Dean Kuga Apr 17 '12 at 16:51
  • @JohnSaunders, following your logic we can conclude that all 7 upvotes in less than 20 minutes are coming from people that own the exact same hardware I do? Are you sure that the question is irrelevant for people who have 32 or 16 GB of RAM on their servers, or run Windows Server 2008 Enterprise, or have their CPUs running at 2.00 GHz instead of 2.27@GHz? – Dean Kuga Apr 17 '12 at 16:59
  • @Marc, are you saying that the number of FileSystemWatcher instances a server can handle is limited only by available resources? If so, what experience or documentation are you relying on please? – Dean Kuga Apr 17 '12 at 17:02
  • @DeanK. I think this is the link that best describes what you're after and comes from MS people directly [FileSystemWatcher across network by Walter Wang MSFT](http://bytes.com/topic/visual-basic-net/answers/536125-filesystemwatcher-across-network#post2092018). And this one [FileSystemWatcher and windows 7](http://stackoverflow.com/a/3250454/417747). That's mostly about the buffer overrun but it's similar, it points to limitation on buffers in system address space and that's created per each + network specifics + medium/heavy suggestions. Still not posting as answer as not clear completely. – NSGaga-mostly-inactive Apr 17 '12 at 17:06
  • @NSGaga, thanks for the link but the InternalBufferSize is a property of an individual FileSystemWatcher instance that can be helpful in situations where a FileSystemWatcher must handle a high volume of nearly simultaneous file system writes. It doesn't directly relate to the question about how many simultensous instances of FileSystemWatcher a system can handle... – Dean Kuga Apr 17 '12 at 17:14
  • It does actually, (as I understood and why I posted it), the buffers (for each) are taken from the kernel memory pools - and there is a buffer on the API call side as well as with file system. So, the amount of memory take accumulates. – NSGaga-mostly-inactive Apr 17 '12 at 17:21

1 Answers1

20

FileSystemWatcher under the cover uses ReadDirectoryChangesW http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx. This is a reasonably inexpensive operation which is just a read from the directory that completes on a change.

The results are stored in a kernel buffer before they are copied into your own memory buffer of FileSystemWatcher.

That's the two OS resources to take into consideration, the Handle created by the call to CreateFile by FileSystemWatcher, and the 8KB (default) buffer size in the Kernel for each FileSystemWatcher object which takes away from your system's Kernel Paged and None-Paged Pools.

Your FileSystemWatchers are essentially competing for these three resources.

  1. CPU time to process the changes
  2. Handles on the system
  3. Page Pool

You're unlikely to hit a problem with (2). Likely to hit a problem with (3) on a power system (loads of CPU) running x86. Otherwise (1) will be your limit.

Handles

Handles are exhaustible (specially on x86), more on this here, http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx

But at 16million+ handles (even on x86) before you run out, for your intententions, I'd think of it as an infinite resource. You'll exhaust the CPU processing changes well before you hit any OS limit.

Page/Non-Paged Pools

Page/Non-Paged Pools can be seen in task manager. On x86 they are very finite. More here, http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx#memory_limits

CPU

You'll see loads of anecdotal evidence that when this is exhausted, FileSystemWatcher sort of stops working. Some directory changes get reported, some don't, and inevitable on large implementations of FileSystemWatcher you end up having to detect these occassions and do a directory listing yourself, or do it on a polling bases.

Notes

If you're implementing a load of FileSystemWatchers watch out for;

  1. Buffer over runs
  2. Buffer size greater than 64KB on network paths.

More on good coding practice for this object here, http://bytes.com/topic/visual-basic-net/answers/536125-filesystemwatcher-across-network#post2092018

M Afifi
  • 4,645
  • 2
  • 28
  • 48
  • 5
    I just want to add on to this question that my experience having created upwards of 5000 FileSystemWatcher objects at a time (combined with polling), I have never run into any major issues with creating huge amounts of FileSystemWatcher objects. – Maxim Gershkovich Nov 23 '20 at 22:28