Spring Integration RecursiveDirectoryScanner gives too many files open exception

Question

I am using Spring Integration RecursiveDirectoryScanner to scan a directory recursively to process the incoming file that will be placed under the configured directory (/home/test).

I am frequently getting the below error:

    ERROR org.springframework.integration.handler.LoggingHandler - java.lang.IllegalArgumentException: java.nio.file.FileSystemException: /home/test: Too many open files
        at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:89)
        at org.springframework.integration.file.FileReadingMessageSource.scanInputDirectory(FileReadingMessageSource.java:387)
        at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:361)
        at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:90)
        at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:134)
        at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:224)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Caused by: java.nio.file.FileSystemException: /home/test: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
        at java.nio.file.Files.newDirectoryStream(Files.java:457)
        at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
        at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
        at java.nio.file.Files.walk(Files.java:3574)
        at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:73)

My Spring Integration flow is as below:

Configuration in XML

 <task:executor id="pollerPool"
        pool-size="${pollerThreadPoolSize}"
        queue-capacity="${pollerThreadQueueCapacity}" rejection-policy="ABORT" />

    <task:executor id="fileHandlerPool"
        pool-size="${fileHandlerPoolSize}"
        queue-capacity="${fileHandlerPoolThreadQueueCapacity}" rejection-policy="CALLER_RUNS" />


     <bean id="iFilter" class="org.springframework.integration.file.filters.ChainFileListFilter">
        <constructor-arg>
             <list>
                  <bean id="lastModifiedFileListFilter" class="org.springframework.integration.file.filters.LastModifiedFileListFilter">
                     <property name="age" value="120" />
                  </bean>
                 <ref bean="acceptOnceFileListFilter"/>

                 <bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
                    <constructor-arg value="^.*\.(txt|csv|xls|xlsx|asc)$"/>
                 </bean>

            </list>
        </constructor-arg>
      </bean>

     <bean id="acceptOnceFileListFilter" name="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" primary="true" />

     <bean id="recursiveDirectoryScanner" class="org.springframework.integration.file.RecursiveDirectoryScanner">
        <property name="filter" ref="iFilter" />
        <property name="locker" ref="nioFileLocker" />
     </bean>

    <bean id="nioFileLocker" class="org.springframework.integration.file.locking.NioFileLocker" />

    <int-file:inbound-channel-adapter
        id="fileSource" channel="fileReceivedChannel" auto-startup="true"
        directory="file:${polling.directory}" 
        scanner="recursiveDirectoryScanner"  >
        <int:poller task-executor="pollerPool"
            fixed-rate="${pollerFixedRate}"
            receive-timeout="${pollerReceiveTimeout}">
        </int:poller>
    </int-file:inbound-channel-adapter>

Dynamic parameters are as below:

polling.directory=/home/test pollerThreadPoolSize=1 pollerThreadQueueCapacity=10 pollerFixedRate=5000 pollerReceiveTimeout=5000 fileHandlerPoolSize=2 fileHandlerPoolThreadQueueCapacity=100

EDIT:

I do unlock file in a service activator that comes in to picture when a file is picked. I get some information from file and unlock it.

@Autowired
  NioFileLocker nioFileLocker;

   protected void doTransform(Message<?> message) throws Exception {
  MessageBuilder<File> payload = (MessageBuilder<File>) message.getPayload();
    File inFile = payload.getPayload();
   try {
      nioFileLocker.unlock(inFile);
    } catch (Exception e) {
      LOGGER.error("file not unlock");
    }
 }

Is there any issue with the configuration ? How do I make sure this exception never appear again ?

Thank you in advance.

Take a look at this: https://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux — ernest_k, Jan 03 '19 at 14:28
Current limit is 1024. By increasing limit I am not sure if it will not happen again. What can be the root cause of the issue ? — mayur tanna, Jan 03 '19 at 14:30
It's just that. Some programs open many files (which is not just files in linux). The defaults aren't always practical for some servers. You probably need to increase the limit, but you just evaluate it before making a decision... — ernest_k, Jan 03 '19 at 14:32
Thanks ernest_k for the quick response. I will check that too. However, if you find any other solution let me know. Thanks a lot. — mayur tanna, Jan 03 '19 at 14:33
Can we see `cause by` of that exception? I wonder which part of the `Files.walk()` is failing... — Artem Bilan, Jan 03 '19 at 14:52

score 0 · Answer 1 · answered Jan 03 '19 at 15:06

0

I would suggest to test your solution without NioFileLocker. Doesn't look like you are using it for unlocking files, but the lock(File fileToLock) really keeps some file marker in OS.

On the other hand the file locker doesn't work reliable on UNIX systems. It still allows an access to files. At least for reading.

For better exclusive file access I would recommend to use a FileSystemPersistentAcceptOnceFileListFilter with the external MetadataStore instead of in memory AcceptOnceFileListFilter. This way only one instance of your application will get access to the file and it won't be processed again at all.

answered Jan 03 '19 at 15:06

Artem Bilan

113,505
11
91
118

I do actually unlock the file. I have added that bit of code also in the question in EDIT section. – mayur tanna Jan 03 '19 at 15:20
How does it work without locker at all? Since your dir is recursive and you have a lot of threads to process, that indeed might be the case that locker is a bottleneck for your solution. – Artem Bilan Jan 03 '19 at 15:35

Spring Integration RecursiveDirectoryScanner gives too many files open exception

1 Answers1