0

I'm writing an application in which I have to detect file changes inside a directory. New files, missing files, and so on. Each scan relative to the previous.

I'm currently doing a recursive scan to retrieve all the paths and comparing to the previous list in my database. The problem with this is that some clients require scanning for millions of files. This makes the system consume a lot of resources (cpu and memory). I'm even getting SystemOutOfMemory exceptions.

So I'm wondering if there's a better way to find those changes, maybe without a full scan.

Important note: I can't "monitor" for events since I have to detect changes that happened between scans, no matter if the system was running. I can't afford to lose a single change. So, unless it can catch changes made while it wasn't running I can't use FileSystemWatcher for this.

Hugo Mota
  • 11,200
  • 9
  • 42
  • 60
  • 2
    Possible duplicate of [Using FileSystemWatcher to monitor a directory](http://stackoverflow.com/questions/15017506/using-filesystemwatcher-to-monitor-a-directory) – Yeldar Kurmangaliyev Aug 17 '16 at 11:59
  • @YeldarKurmangaliyev In the final paragraph I explained that I can't **monitor** for changes since the system may not be running all the time. – Hugo Mota Aug 17 '16 at 12:02
  • http://devproconnections.com/net-framework/how-build-folder-watcher-service-c – Dah Sra Aug 17 '16 at 12:10
  • FileSystemWatcher is just not reliable enough for my use case. I can't miss a single change. – Hugo Mota Aug 17 '16 at 12:17

1 Answers1

1

Without knowing how the Directory you are scanning is structured im assuming you have something simmilar to following example.

000/
001/
002/

With each directory having subdirectories simmilar to the top level directory.

This would allow you to build up an index like git does internally. Storing identifiers for each tree item (subdirectories, Files) for easy comparrison.

You should then be able to counter the SystemOutOfMemoryException through splitting up the task into multiple sub tasks.

Regarding the runtime i see no possibility to lower it as one component of your system will always have to watch or compare each Item.

If when the files are written the modify date of the directories ist reliably updated you could use this as part of your comparrison.