Best way to monitor huge amount of files and dynamic directories in Windows 10

Question

I am making a desktop application that monitors a directory that contains dynamic sub-directories and files. There are a lot of files(logs) being created in these sub-directories. I want to make an application that watches these files being created (around 50 files created within 5 mins) notify.

I went through the WatchDir provided by java.nio, but as we need to register a watchkey for each directory, I fear I might run into some concurrency issues later on.

I tried searching for alternatives and I found monitor by Apache Commons, but it takes up a lot of CPU as per this answer. In my case I want to monitor large amount of files but they are also not continuous. I want to watch files only for a few hours in a day and during this time there will be a lot of files generated. What would be the best option here?

Can you explain what you mean by "I fear I might run into some concurrency issues later on"? — tgdavies, Feb 13 '21 at 03:38
Does this answer your question? [list all files in the folder and also sub folders](https://stackoverflow.com/questions/14676407/list-all-files-in-the-folder-and-also-sub-folders) — Md Kawser Habib, Feb 13 '21 at 05:42

score 1 · Answer 1 · answered Feb 13 '21 at 04:00

If you're concerned about how many threads watching uses, it's easy to write a small program to check:

import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;

import static java.nio.file.StandardWatchEventKinds.ENTRY_CREATE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_DELETE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;

public class Example {
    public static void main(String... args) throws IOException, InterruptedException {
        final WatchService watcher = FileSystems.getDefault().newWatchService();
        final int number_of_watches = 100;
        final Path[] paths = new Path[number_of_watches];
        final WatchKey[] watchKeys = new WatchKey[number_of_watches];
        final Path tmp = Files.createTempDirectory("watch");
        for (int i = 0; i < number_of_watches; i++) {
            paths[i] = tmp.resolve("dir" + i);
            Files.createDirectory(paths[i]);
            watchKeys[i] = paths[i].register(watcher,
                    ENTRY_CREATE,
                    ENTRY_DELETE,
                    ENTRY_MODIFY);
            System.out.println(paths[i]);
        }
        Thread.getAllStackTraces().keySet().stream().forEach(t -> System.out.println(t));
        
        for (;;) {
            WatchKey key = watcher.take();
            System.out.println("Change in " + key.watchable());
        }
    }
}

The output ion the thread name part is:

Thread[Common-Cleaner,8,InnocuousThreadGroup]
Thread[main,5,main]
Thread[Finalizer,8,system]
Thread[FileSystemWatcher,5,main]
Thread[Signal Dispatcher,9,system]
Thread[Reference Handler,10,system]
Thread[Notification Thread,9,system]

So this is using only one thread, Thread[FileSystemWatcher,5,main]

Best way to monitor huge amount of files and dynamic directories in Windows 10

1 Answers1