Track file import into database

Question

I need to test file import status using this Java code:

    @EventListener
    public void handleContextStart(ContextRefreshedEvent eventd) throws IOException, InterruptedException {
        System.out.println("Handling context started event.!!!");

        System.out.println("Running file verifier");
        System.out.println("monitoring folder " + folderPath);

        WatchService watchService = FileSystems.getDefault().newWatchService();
        Path path = Paths.get(folderPath);
        path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_DELETE, StandardWatchEventKinds.ENTRY_MODIFY);
        WatchKey key;
        while ((key = watchService.take()) != null) {
            for (WatchEvent<?> event : key.pollEvents()) {
                System.out.println("Event kind:" + event.kind() + ". File affected: " + event.context() + ".");
                if(event.kind().equals(StandardWatchEventKinds.ENTRY_DELETE)){
                    Instant start = Instant.now();

                    AtomicBoolean flag = new AtomicBoolean(true);

                    while(flag.get()) {
                        while ((key = watchService.take()) != null) {

                            List<EntityImportRequestsTable> list = entityImportRequestsService.findAll();
                            HashMap<String, String> map = new HashMap<>();
                            for(EntityImportRequestsTable item : list){
                                map.put(item.getRequestXmlSourceFile(), item.getStatusCode());
                            }

                            map.entrySet().forEach(entry -> {
                                System.out.println(entry.getKey() + " " + entry.getValue());

                                // compare file name from list and file name from the delete event by file name
                                if(entry.getKey().contains(event.context().toString())){
                                    // exit the monitoring while loop
                                    flag.set(false);
                                }
                            });
                        }
                        Thread.sleep(1000);
                    }

                    Instant end = Instant.now();
                    System.out.println(Duration.between(start,end));

                    long seconds = TimeUnit.MILLISECONDS.toSeconds(Duration.between(start,end).getSeconds());
                    long minutes = TimeUnit.MILLISECONDS.toMinutes(Duration.between(start,end).getSeconds());

                    System.out.format("Execution time %d minutes %d seconds", minutes, seconds);


                }

            }
            key.reset();
        }

        watchService.close();
    }

When file is processed from the tested app the file is deleted and imported into database. If the file import is successfully imported the status is completed.

When I run the above program noting happens. Do you know how I can properly track the file import?

Answer 1

Your loop over the events is flawed as it will skip over events because you have double calls to watchService.take(). The second call may block, and if it works the events will be skipped so the inner loop may never match up a delete or exit correctly. Remove the second take():

while ((key = watchService.take()) != null) {
   ... 
   while ((key = watchService.take()) != null) {
   ... 

Here is an example of logic for setting up the WatchService, you should try with System.out.println() to understand how it works:

Path watchDir = Path.of("whatever");
try (WatchService service = watchDir.getFileSystem().newWatchService()) {

    // Add whatever StandardWatchEventKinds events you need
    WatchKey wk = watchDir.register(service, StandardWatchEventKinds.ENTRY_DELETE /*, ENTRY_CREATE ENTRY_DELETE ... ETC */);
    // Insert your own loop termination condition here:
    while (running) {
        WatchKey key = service.poll(1, TimeUnit.SECONDS);
        if (key == null) {
            continue;
        }

        for (var event : key.pollEvents())  {
            Path evPath  = (Path) event.context();
            Path absPath = ((Path) key.watchable()).resolve(evPath);
            Kind<?> kind = event.kind();

            System.out.println("Kind=" + kind + " Path=" + evPath+" AbsPath=" + absPath);
            // Check each event to see if your termination condition is met
        }

        key.reset();
    }
}

Running the above should give you an idea of what WatchService provides, and you can fill in your own logic into pollEvents() loop to handle each delete notification:

 // Check each event to see if your termination condition is met
 if (kind == StandardWatchEventKinds.ENTRY_DELETE) {
     if (absPath is one of the files you want) {
         Mark as received + print timings
         running = (number of remaining files you want > 0);
     }
 }

You may find it easier to refactor the watch service part as a separate component so that it collates events and delivers a set of changes to a listener - see example here.

If your listener/handler inside the loop can take some time, run watch service and handler in separate threads so that the event loop keeps up with watch event publishing, and don't use sleep as that may cause overflow events because your listener/handler is too slow.