2

Problem description
I would like to watch a complete file system for changes. I'm talking about watching changes in a directory recursively. So, when watching a directory (or a whole file system) all changes in sub-directories need to be captured too. The application needs to be able to track all changes by getting notified.

Java's WatchService isn't suitable
Java already has a WatchService feature, which allows you to monitor a directory for changes. The problem is however, that this isn't a recursive process as far as I know, thus you can't use this to monitor all changes in the root directory of a file system.

Watching all sub-directories explicitly
A solution I've thought of would be to register each directory inside the specified root directory explicitly. The problem with this is however, that walking through, and registering these directories is very resource expensive on a system with more than a million sub-directories. This is because the system would need to go through the whole file system recursively to only register all directories in the first place. The performance impact of this feature would be too big, if it's even possible without crashing the application.

Logical solution
I would assume an operating system would fire/call some sort of event when anything is changed on the file system, that an application is able to listen to. I did however, not find anything like this yet. This would allow the application to listen to all changes without the need to register all sub-directories explicitly. Thus the performance impact with such a method would be minimal.

Question
Is watching a whole file system, or watching a directory recursively possible in Java, and how would this be achieved?

Tim Visée
  • 2,988
  • 4
  • 45
  • 55
  • Possible duplicate of http://stackoverflow.com/questions/18701242/how-to-watch-a-folder-and-subfolders-for-changes – Sheetal Mohan Sharma Jan 25 '16 at 15:38
  • The duplicate you mentioned only seems to be directed to the `WatchService`. This question is about watching a file system in general, the `WatchService` mentioned is just to cover a method that isn't suitable. – Tim Visée Jan 25 '16 at 15:40
  • 1
    I would say that this is probably impossible for a directory with millions of subdirectories. At least not on Linux, where this monitoring is done by `inotify`, which is *not recursive* for the [reasons given on Quora by the author](https://www.quora.com/Inotify-monitoring-of-directories-is-not-recursive-Is-there-any-specific-reason-for-this-design-in-Linux-kernel). – RealSkeptic Jan 25 '16 at 16:04

3 Answers3

2

The question should be split into several:

  1. How to track file events across the disk on certain OS
  2. How to use this mechanism in Java

The answer to the first question is that the approaches are different. On Windows there exist Windows API functions that let you do this (and famous FileSystemWatcher class in .NET Framework is a kind of wrapper around this API function set). The more robust method on windows is to create or use a pre-created file system filter driver. On Linux there exists inotify. On MacOS X there exist several approaches (there was a question on this topic somewhere around), none of them being universal or always available. Also all approaches except a filesystem filter driver are good only for being notified after the event happens, but they don't let you intercept and deny the request (AFAIK, I can be mistaken here).

As for the second question, there seems to be no universal solution that would cover all or most variants that I mentioned above. You would need to first choose the mechanism for each OS, then find some wrappers for Java to use those mechanisms.

Eugene Mayevski 'Callback
  • 45,135
  • 8
  • 71
  • 121
0

Here is an example to to watch a directory (or tree) for changes to file

Sheetal Mohan Sharma
  • 2,908
  • 1
  • 23
  • 24
  • 1
    This method uses the method I mentioned in my question. It is registering all sub-directories explicitly as you can see in the `registerAll()` method. This just wouldn't be an option for a file system that has more than a million directories. – Tim Visée Jan 25 '16 at 15:41
  • Ah..well that is a complicated situation then? Cant you create a cache and update that cache in separate thread say every 30 mins? – Sheetal Mohan Sharma Jan 25 '16 at 15:45
  • Do you mean, cache all files in the file system, and compare the cache to the file system again after 30 minutes? No, this would require the application to go through the whole file system again every 30 minutes. Going through everything once might take up to 12 hours on a HDD storage device. – Tim Visée Jan 25 '16 at 15:47
  • 7
    Your answer is a "link only" answer which is not considered a proper answer on Stack Overflow. You should explain the gist of the link's content in the answer itself. Links may be broken. At the very least you should mention what the link is linking to. – RealSkeptic Jan 25 '16 at 15:48
0

Please find example https://github.com/syncany/syncany/blob/59cf87c72de4322c737f0073ce8a7ddd992fd898/syncany-lib/src/main/java/org/syncany/operations/watch/RecursiveWatcher.java

Even you can filtered our directory that you don't want to watch

Pankaj Pandey
  • 1,015
  • 6
  • 10
  • 1
    This answer is similar to the one *Sheetal Mohan Sharma* has posted. The problem with this example is that the application is going through all the directories on the file system, and it registers them explicitly. This would require the application to register more than a million directories, which isn't a suitable solution. The performance impact of this method would be huge, making the application unusable. – Tim Visée Jan 25 '16 at 15:59
  • This is not the answer. the question was - without explicitly watching each folder in a tree. On a disk with millions folders this is impossible – sergeych Sep 01 '23 at 20:30