0

I've built a HotFolderWatcher class, based on FileSystemWatcher, which I use in my application. Unfortunately, our automated units lend support to the complaints I've read online that FileSystemWatcher itself isn't reliable -- that is, it is only as reliable as the OS' handling of the underlying events (and this isn't reliable).

Is there any technology or solution to have a reliable hot folder on Windows?

gap
  • 2,766
  • 2
  • 28
  • 37
  • 1
    I never had problem with FileSystemWatcher... did you actually experienced a problem with it or are you worry about a random comment on the web? – The_Black_Smurf Jul 16 '15 at 14:29
  • "only as reliable as the OS" This implies that the OS in question, Windows I take it isn't reliable. No technology is guaranteed to never break and work 100% of the time with absolutely no errors, if windows is not reliable enough what are they using it for? Reliability of the class part of things, the class wouldn't be there if it wasn't reliable unless you got it from a third party, the class has an intended scenario under which it is intended for use, did you see if that is your case? – CalebB Jul 16 '15 at 14:30
  • @The_Black_Smurf - yes, our automated unit tests reveal intermittent failures in the detection of newly created files. When it occurs, we only miss 1 file. – gap Jul 16 '15 at 14:37
  • 1
    @Caleb, The_Black_Smurf, FileSystemWatcher and the API it relies on **is** unreliable. It tends to miss event, especially in high volume situations. See [FileSystemWatcher vs polling to watch for file changes](http://stackoverflow.com/questions/239988/), [FileSystemWatcher reliability](https://social.msdn.microsoft.com/Forums/vstudio/en-US/4465cafb-f4ed-434f-89d8-c85ced6ffaa8/filesystemwatcher-reliability) and so on. – CodeCaster Jul 16 '15 at 14:37
  • I've always found FileSystemWatcher reliable on a local file system but not so across a network. in the past I've written a class that just crawls a directory and raises event when new file identified. Not great on IO, so interested to see if anyone has a better answer. – GinjaNinja Jul 16 '15 at 14:42

1 Answers1

3

One way would be to use FileSystemWatcher to handle most of the cases, and on an interval use Directory.GetFiles to get the real list of files.

That way you have the efficiency of FileSystemWatcher to handle most cases where it will work, and you can be sure that other changes will be caught on the interval you specify.

Edit: of course Directory.GetFiles won't tell you about renames or changes to a file, so you would need additional logic to detect these events if you need them.

David Thibault
  • 8,638
  • 3
  • 37
  • 51
  • The system needs to be handle MANY files... perhaps 1k directories of 10k files each. I'm hesitant to call GetFiles at all, let alone more than once. – gap Jul 16 '15 at 14:35
  • I see. The calls to GetFiles could run in a background thread so that it does not block the rest of your application. – David Thibault Jul 16 '15 at 14:40
  • 1
    You could also take a look at `DirectoryInfo.EnumerateFiles`, which could be more efficient. – David Thibault Jul 16 '15 at 14:40
  • To be 100% reliable you need some sort of checking which files already have been processed and which not. Therefore you need to sync the actual files in the folder/s with some kind of data source (DB probably). This will be no problem even for a huge amount of folders, since this is a fallback you can do this every hour or less if it takes too long. – Denis Thomas Jul 16 '15 at 14:41