0

I created simple job which reads all files from folder(D:\\chunk), does empty procesing and empty writing and I registered listener to remove file after processing.

On Windows(On Linux and MacOs it does not happen) mashine I experince following error:

2019-09-09 12:08:13.752  WARN 4028 --- [           main] c.b.m.b.RemovingListener   : Failed to remove chunk 0b9a2623-b4c3-42b2-9acf-373a2d81007c.csv

java.nio.file.FileSystemException: D:\chunk\1.csv: The process cannot access the file because it is being used by another process.

    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
    at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:270)
    at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
    at java.base/java.nio.file.Files.deleteIfExists(Files.java:1180)
    at my.super.project.batch.RemovingListener.afterStep(RemovingListener.java:31)
    at my.super.project.batch.RemovingListener$$FastClassBySpringCGLIB$$e695a1e2.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:750)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
    at my.super.project.batch.RemovingListener$$EnhancerBySpringCGLIB$$10d47ff9.afterStep(<generated>)

Let me show you source:

I start job like this(java 11 syntax):

Path location = Path.of("D:\\chunk");
var parameters = new JobParametersBuilder()
        .addString("files.location", location.toUri().resolve("*").toString()).toJobParameters();

jobLauncher.run(job, parameters);

job configuration:

@Bean
public Job fileProcessingJob(
        MultiResourcePartitioner partitioner,
        Step slaveStep
) {
    return jobBuilderFactory
            .get("read-file-job")
            .start(stepBuilderFactory.get("master-step")
                    .partitioner("processChunk", partitioner)
                    .step(slaveStep)
                    .build())
            .build();
}

partitioner:

@Bean
@JobScope
public MultiResourcePartitioner filesPartitioner(
        ResourcePatternResolver resolver,
        @Value("#{jobParameters['files.location']}") String location
) throws IOException {
    var partitioner = new MultiResourcePartitioner();
    partitioner.setResources(resolver.getResources(location));
    return partitioner;
}

slave step:

@Bean
    public Step slaveStep(
            FlatFileItemReader<String> reader,
            RemovingListener listener
    ) {
        return stepBuilderFactory.get("processChunk")
                .<String, String>chunk(10)
                .reader(reader)
                .processor((Function<String, String>) s -> s) //empty
                .writer(items -> { //empty
                })
                .listener(listener)
                .build();
    }

RemovingListener:

@StepScope
@Component
public class RemovingListener extends StepExecutionListenerSupport {

    private final Resource resource;

    public RemovingListener(@Value("#{stepExecutionContext['fileName']}") Resource resource) {
        this.resource = resource;
    }

    @Override
    public ExitStatus afterStep(@NonNull StepExecution stepExecution) {
        try {
            Files.deleteIfExists(resource.getFile().toPath());
        } catch (IOException e) {
            log.warn("Failed to remove chunk {}", resource.getFilename(), e);
        }
        return stepExecution.getExitStatus();
    }
} 

What is it going on?

Why does it happen ? how to fix it ?

Full source could be found here: https://github.com/gredwhite/spring-batch-hello-world/tree/master/src/main/java/spring/boot/hello/process_cannot_access

gstackoverflow
  • 36,709
  • 117
  • 359
  • 710
  • 1
    There must be another process using the file (have you run the job multiple times due to a failure (and a JVM is still using the file)?) You need to kill that process before relaunching your job (This SO thread might help: https://stackoverflow.com/questions/379689/identify-process-using-a-file). – Mahmoud Ben Hassine Sep 09 '19 at 10:17
  • @Mahmoud Ben Hassine I checked this version and it is not my case. This file is used only by this process. 100% Moreover my investigation showed that if to use listener on level of job - everything is working properly – gstackoverflow Sep 09 '19 at 10:36
  • @Mahmoud Ben Hassine if you have a windows mashine you can run the code I provided – gstackoverflow Sep 09 '19 at 11:33
  • @Mahmoud, Right now I've restarted PC to be sure that no other processes use files from my folder and run the application and I see the same error so it is a prove that the problem inside the application. Could I ask you to check it ? – gstackoverflow Sep 09 '19 at 11:55
  • If it works on linux/mac but fails on MS windows, I assume the problem is not with the code. BTW, I see nothing wrong in your code. – Mahmoud Ben Hassine Sep 09 '19 at 12:49
  • The problem is there on Unix system as well. We don't see it there because Windows is too sensitive to resources which are not closed before delete/rename. There should be something which opens `InputStream/OutputStream` for the file and doesn't close it after reading/writing. So, I guess we need to see your ` .writer(items -> {` code or something else. i think you need to debug your application where that `D:\chunk\1.csv` is opened form streaming, but is not closed in the end, before removing. – Artem Bilan Sep 09 '19 at 13:01
  • I think we even are good with @MahmoudBenHassine to debug this for you if you can share with us a simple project to play with. – Artem Bilan Sep 09 '19 at 13:02
  • @Artem Bilan actually it is full code. I made empty writer on purpose. Let me som etime to create separated repo with that code. – gstackoverflow Sep 09 '19 at 14:29
  • @Mahmoud Ben Hassine are streams closed before the call to `afterStep()`? maybe create a new step to delete files after use could help – Luca Basso Ricci Sep 09 '19 at 14:31
  • @Artem Bilan, please take a look https://github.com/gredwhite/spring-batch-hello-world/tree/master/src/main/java/spring/boot/hello/process_cannot_access – gstackoverflow Sep 09 '19 at 15:20
  • @Artem Bila, Did you have a chance to take a look ? – gstackoverflow Sep 10 '19 at 08:32
  • @Mahmoud Ben Hassine, Did you have a chance to take a look ? – gstackoverflow Sep 10 '19 at 08:38
  • **open** : org.springframework.batch.core.step.AbstractStep#execute 200 **listener invocation** : org.springframework.batch.core.step.AbstractStep#execute 242 **close** : org.springframework.batch.core.step.AbstractStep#execute 274 – gstackoverflow Sep 10 '19 at 09:10
  • @Mahmoud Ben Hassine, Is it correct place? – gstackoverflow Sep 10 '19 at 09:10
  • @Mahmoud Ben Hassine I believe that streams are closed AFTER afterStep invocation but it should be before – gstackoverflow Sep 10 '19 at 09:12
  • listener invocation: https://github.com/spring-projects/spring-batch/blob/4d3090ac023d1ebc9e95ef735378647b7db74f7e/spring-batch-core/src/main/java/org/springframework/batch/core/step/AbstractStep.java#L247 – gstackoverflow Sep 10 '19 at 09:27
  • stream closing: https://github.com/spring-projects/spring-batch/blob/4d3090ac023d1ebc9e95ef735378647b7db74f7e/spring-batch-core/src/main/java/org/springframework/batch/core/step/AbstractStep.java#L286 – gstackoverflow Sep 10 '19 at 09:28
  • @Mahmoud Ben Hassine, Actually I have no ideas how to create issue on github without pull request. – gstackoverflow Sep 10 '19 at 09:35
  • Streams are indeed closed after `StepExecutionListener#afterStep`. My bad sorry for that. So I guess this is the cause of your issue. However I'm not sure if this is a bug or a documentation issue. Probably a `JobExecutionListener` is more appropriate for this use case. In regards to creating issues, you can do it on the [JIRA instance](https://jira.spring.io/projects/BATCH/summary) of the project. – Mahmoud Ben Hassine Sep 10 '19 at 11:17
  • 1
    Grate! Looks we are agreed that there is bug in Spring Batch. I would say it an abuse to move logic to the job listener. IMO streams have to be closed before `afterStep`. It is step scope to deal with those streams – Artem Bilan Sep 10 '19 at 12:23
  • @Artem Bilan, created bug - https://jira.spring.io/browse/BATCH-2842 . Feel free to modify if something wrong – gstackoverflow Sep 10 '19 at 13:07
  • 1
    I agree, using a job listener or another step (as suggested by Luca) is more like working around problem. Probably there is a reason why the listener is called before closing streams, but I don't have enough context to answer this question for now. Thank you @gstackoverflow for opening the issue. We will look at this and get back to you. – Mahmoud Ben Hassine Sep 10 '19 at 13:10
  • For now I decided to create an additional step(tasklet) to remove file and it works correctly – gstackoverflow Sep 10 '19 at 13:42

0 Answers0