18

I wan't to write Spring Boot Application in spring which will be monitoring directory in windows, and when I change sub folder or add new one or delete existing one I wanna get information about that.

How can i do that? I have read this one: http://docs.spring.io/spring-integration/reference/html/files.html and each result under 'spring file watcher' in google, but I can't find solution...

Do you have a good article or example with something like this? I wan't it to like like this:

@SpringBootApplication
@EnableIntegration
public class SpringApp{

    public static void main(String[] args) {
        SpringApplication.run(SpringApp.class, args);
    }

    @Bean
    public WatchService watcherService() {
        ...//define WatchService here
    }
}

Regards

amkz
  • 568
  • 3
  • 9
  • 31
  • You should start from the `FileSystemWatcher` class and then add `FileChangeListener`(s). Or you can use the `WatchService` introduced with Java 7: http://andreinc.net/2013/12/06/java-7-nio-2-tutorial-writing-a-simple-filefolder-monitor-using-the-watch-service-api/ – Andrei Ciobanu Oct 13 '16 at 12:34

8 Answers8

18

spring-boot-devtools has FileSystemWatcher

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
</dependency>

FileWatcherConfig

@Configuration
public class FileWatcherConfig {
    @Bean
    public FileSystemWatcher fileSystemWatcher() {
        FileSystemWatcher fileSystemWatcher = new FileSystemWatcher(true, Duration.ofMillis(5000L), Duration.ofMillis(3000L));
        fileSystemWatcher.addSourceFolder(new File("/path/to/folder"));
        fileSystemWatcher.addListener(new MyFileChangeListener());
        fileSystemWatcher.start();
        System.out.println("started fileSystemWatcher");
        return fileSystemWatcher;
    }

    @PreDestroy
    public void onDestroy() throws Exception {
        fileSystemWatcher().stop();
    }
}

MyFileChangeListener

@Component
public class MyFileChangeListener implements FileChangeListener {

    @Override
    public void onChange(Set<ChangedFiles> changeSet) {
        for(ChangedFiles cfiles : changeSet) {
            for(ChangedFile cfile: cfiles.getFiles()) {
                if( /* (cfile.getType().equals(Type.MODIFY) 
                     || cfile.getType().equals(Type.ADD)  
                     || cfile.getType().equals(Type.DELETE) ) && */ !isLocked(cfile.getFile().toPath())) {
                    System.out.println("Operation: " + cfile.getType() 
                      + " On file: "+ cfile.getFile().getName() + " is done");
                }
            }
        }
    }
    
    private boolean isLocked(Path path) {
        try (FileChannel ch = FileChannel.open(path, StandardOpenOption.WRITE); FileLock lock = ch.tryLock()) {
            return lock == null;
        } catch (IOException e) {
            return true;
        }
    }

}
Sully
  • 14,672
  • 5
  • 54
  • 79
  • 1
    Be warned... https://docs.spring.io/spring-boot/docs/current/reference/html/using.html#using.devtools.remote-applications – David Dec 03 '21 at 22:19
  • 1
    @David, excellent point. The feature is used mainly for hot-deploying. I would strip everything else from the import – Sully Dec 05 '21 at 18:20
16

From Java 7 there is WatchService - it will be the best solution.

Spring configuration could be like the following:

@Slf4j
@Configuration
public class MonitoringConfig {

    @Value("${monitoring-folder}")
    private String folderPath;

    @Bean
    public WatchService watchService() {
        log.debug("MONITORING_FOLDER: {}", folderPath);
        WatchService watchService = null;
        try {
            watchService = FileSystems.getDefault().newWatchService();
            Path path = Paths.get(folderPath);

            if (!Files.isDirectory(path)) {
                throw new RuntimeException("incorrect monitoring folder: " + path);
            }

            path.register(
                    watchService,
                    StandardWatchEventKinds.ENTRY_DELETE,
                    StandardWatchEventKinds.ENTRY_MODIFY,
                    StandardWatchEventKinds.ENTRY_CREATE
            );
        } catch (IOException e) {
            log.error("exception for watch service creation:", e);
        }
        return watchService;
    }
}

And Bean for launching monitoring itself:

@Slf4j
@Service
@AllArgsConstructor
public class MonitoringServiceImpl {

    private final WatchService watchService;

    @Async
    @PostConstruct
    public void launchMonitoring() {
        log.info("START_MONITORING");
        try {
            WatchKey key;
            while ((key = watchService.take()) != null) {
                for (WatchEvent<?> event : key.pollEvents()) {
                    log.debug("Event kind: {}; File affected: {}", event.kind(), event.context());
                }
                key.reset();
            }
        } catch (InterruptedException e) {
            log.warn("interrupted exception for monitoring service");
        }
    }

    @PreDestroy
    public void stopMonitoring() {
        log.info("STOP_MONITORING");

        if (watchService != null) {
            try {
                watchService.close();
            } catch (IOException e) {
                log.error("exception while closing the monitoring service");
            }
        }
    }
}

Also, you have to set @EnableAsync for your application class (it configuration).

and snipped from application.yml:

monitoring-folder: C:\Users\user_name

Tested with Spring Boot 2.3.1.


Also used configuration for Async pool:

@Slf4j
@EnableAsync
@Configuration
@AllArgsConstructor
@EnableConfigurationProperties(AsyncProperties.class)
public class AsyncConfiguration implements AsyncConfigurer {

    private final AsyncProperties properties;

    @Override
    @Bean(name = "taskExecutor")
    public Executor getAsyncExecutor() {
        log.debug("Creating Async Task Executor");
        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        taskExecutor.setCorePoolSize(properties.getCorePoolSize());
        taskExecutor.setMaxPoolSize(properties.getMaxPoolSize());
        taskExecutor.setQueueCapacity(properties.getQueueCapacity());
        taskExecutor.setThreadNamePrefix(properties.getThreadName());
        taskExecutor.initialize();
        return taskExecutor;
    }

    @Bean
    public TaskScheduler taskScheduler() {
        return new ConcurrentTaskScheduler();
    }

    @Override
    public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
        return new CustomAsyncExceptionHandler();
    }
}

Where the custom async exception handler is:

@Slf4j
public class CustomAsyncExceptionHandler implements AsyncUncaughtExceptionHandler {

    @Override
    public void handleUncaughtException(Throwable throwable, Method method, Object... objects) {
        log.error("Exception for Async execution: ", throwable);
        log.error("Method name - {}", method.getName());
        for (Object param : objects) {
            log.error("Parameter value - {}", param);
        }
    }
}

Configuration at properties file:

async-monitoring:
  core-pool-size: 10
  max-pool-size: 20
  queue-capacity: 1024
  thread-name: 'async-ex-'

Where AsyncProperties:

@Getter
@Setter
@ConfigurationProperties("async-monitoring")
public class AsyncProperties {
    @NonNull
    private Integer corePoolSize;
    @NonNull
    private Integer maxPoolSize;
    @NonNull
    private Integer queueCapacity;
    @NonNull
    private String threadName;
}

For using asynchronous execution I am processing an event like the following:

validatorService.processRecord(recordANPR, zipFullPath);

Where validator service has a look like:

@Async
public void processRecord(EvidentialRecordANPR record, String fullFileName) {

The main idea is that you configure async configuration -> call it from MonitoringService -> put @Async annotation above method at another service which you called (it should be a method of another bean - initialisation goes through a proxy).

catch23
  • 17,519
  • 42
  • 144
  • 217
  • Control will never return from below method cause it will stuck in endless loop here and spring context will never load. @PostConstruct public void launchMonitoring() { while ((key = watchService.take()) != null) – Rayon Sep 09 '20 at 12:44
  • @Rayon fixed it with adding `@Async`. – catch23 Sep 09 '20 at 13:05
  • @catch23 I tried this with Async annotation, the control doesn't seem to return. – Jay Nov 24 '20 at 15:18
  • @Jay for me it returns fine. However, I have additional configuration for async pool - `AsyncConfiguration implements AsyncConfigurer`. Actually it shouldn't return. It should listen to events for a folder and process it at asynchronous thread. – catch23 Nov 24 '20 at 19:51
  • 1
    @catch23 could you please upload the project code to github? I tried the AsyncConfiguration but it made no difference. My code gets stuck in the while loop and the spring controllers don't seem to start, so none of the rest controllers are accessbile. – Jay Nov 25 '20 at 21:57
  • @Jay project is very heavy. It is small piece from it. There are too much redundant stuff – catch23 Nov 25 '20 at 22:00
  • @catch23 could you look into my code here - https://github.com/JayVem/temp and please tell me why it might not be working – Jay Nov 25 '20 at 22:25
  • @catch23 you can check sample [here](https://stackoverflow.com/a/66155158/8586437) – Rayon Feb 11 '21 at 12:54
  • 1
    This is a great solution, I just adapted it to launch a new thread for each directory that I need to monitor... realistically it shouldn't be more than 2 directories in our case. I found that putting a slight Thread.sleep() of 3000 ms between the watchService.take() and key.pollEvents() calls prevents duplicate ENTRY_MODIFY events being fired (1 for the file content and 1 for the files modified-date) – MattWeiler Feb 06 '22 at 00:42
  • Hey, I'm using the exact same solution you proposed but the asynchronous execution does not work... – thmasker Jan 17 '23 at 11:33
  • @thmasker I used it in the same way as it is described. Added few more details to the answer. – catch23 Jan 17 '23 at 16:31
5

You can use pure java for this no need for spring https://docs.oracle.com/javase/tutorial/essential/io/notification.html

Sławomir Czaja
  • 283
  • 1
  • 7
  • Yes i know, but I want to use spring because after that I can for example print the result on webpage using websockets, or something else... – amkz Oct 13 '16 at 12:48
  • 1
    @AdamKortylewicz Then use Spring for the web part, but what this answer is telling you is that there's nothing Spring-specific (or even related) in your question, and that the solution is to use a feature existing in core Java. – kaqqao Oct 13 '16 at 13:09
  • 2
    This is true, however, today we're monitoring a local directory and tomorrow we need to analyze a remote directory. Maybe it's a bucket on AWS or some other cloud provider. Using Spring Integration one *could* argue that these details are abstracted away more cleanly – IcedDante Sep 18 '18 at 18:17
  • The java WatchService does not work well for remote file systems, especially if NFS is used on the remote server – Victor Marrerp Aug 08 '20 at 08:40
  • 1
    @IcedDante if you want to monitor S3, use Lambda and S3 events https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html – Sully Nov 13 '20 at 15:26
2

See the Spring Integration Samples Repo there's a file sample under 'basic'.

There's a more recent and more sophisticated sample under applications file-split-ftp - it uses Spring Boot and Java configuration Vs. the xml used in the older sample.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
1

found a workaround you can annotate your task by @Scheduled(fixedDelay = Long.MAX_VALUE)

you could check code:

@Scheduled(fixedDelay = Long.MAX_VALUE)
public void watchTask() {
                this.loadOnStartup();
                try {
                    WatchService watcher = FileSystems.getDefault().newWatchService();
                    Path file = Paths.get(propertyFile);
                    Path dir = Paths.get(file.getParent().toUri());
                    dir.register(watcher, ENTRY_MODIFY);
                    logger.info("Watch Service registered for dir: " + dir.getFileName());
    
                    while (true) {
                        WatchKey key;
                        try {
                            key = watcher.take();
                        } catch (InterruptedException ex) {
                            return;
                        }
    
                        for (WatchEvent<?> event : key.pollEvents()) {
                            WatchEvent.Kind<?> kind = event.kind();
    
                            @SuppressWarnings("unchecked")
                            WatchEvent<Path> ev = (WatchEvent<Path>) event;
                            Path fileName = ev.context();
                            logger.debug(kind.name() + ": " + fileName);
                            if (kind == ENTRY_MODIFY &&
                                    fileName.toString().equals(file.getFileName().toString())) {
                                    //publish event here
                            }
                        }
                        boolean valid = key.reset();
                        if (!valid) {
                            break;
                        }
                    }
                } catch (Exception ex) {
                    logger.error(ex.getMessage(), ex);
                }
            }
        }
catch23
  • 17,519
  • 42
  • 144
  • 217
Rayon
  • 661
  • 8
  • 9
0

Without giving the details here a few pointers which might help you out.

You can take the directory WatchService code from Sławomir Czaja's answer:

You can use pure java for this no need for spring https://docs.oracle.com/javase/tutorial/essential/io/notification.html

and wrap that code into a runnable task. This task can notify your clients of directory change using the SimpMessagingTemplate as described here: Websocket STOMP handle send

Then you can create a scheduler like described here: Scheduling which handles the start and reaccurance of your task.

Don't forget to configure scheduling and websocket support in your mvc-config as well as STOMP support on the client side (further reading here: STOMP over Websocket)

s.ijpma
  • 930
  • 1
  • 11
  • 23
  • So how can I make WatchService a @Bean ? Because I wan't to create method which return WatchService as bean – amkz Oct 13 '16 at 13:33
  • You could, but to have a more manageble approach I would use a scheduler that triggers the WatchService task. – s.ijpma Oct 13 '16 at 13:46
  • @amkz have a look at my answer. There is a configuration for making `WatchService` as a Spring bean. – catch23 Aug 05 '20 at 10:31
0

Apache commons-io is another good alternative to watch changes to files/directories.

You can see the overview of pros and cons of using it in this answer: https://stackoverflow.com/a/41013350/16470819

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.11.0</version>
</dependency>
-1

Just in case, if somebody is looking for recursive sub-folder watcher, this link may help: How to watch a folder and subfolders for changes

Sanket Mehta
  • 622
  • 6
  • 10