Spring batch pass value of resource from step 1 to the next step

Question

So my spring batch job has three steps, the first step(tasklet) will copy files from a shared drive and store them in a folder of the spring project(FileSystems.getDefault()."folder_name"). The second step should read the xml file from this folder and process as per business logic and finally write into the database.The final step(tasklet) will delete the folder created in step 1. But I am getting "java.lang.IllegalArgumentException: Resources must be set and o.s.batch.core.step.AbstractStep : Exception while closing step execution resources in step step1 in job job java.lang.NullPointerException: null." I am using MultiResourceItemReader to read the files in the second step. Can anybody help on how to remove this exception and so that the batch executes as per the mentioned logic.

This is the batch configuration-

@Autowired
public TaskletStep taskletStep;

@Autowired
public MyItemReader myItemReader;

@Autowired
public MyItemWriter myItemWriter;

@Autowired
public JobBuilderFactory jobBuilderFactory;

@Autowired
public StepBuilderFactory stepBuilderFactory;

@Bean
public Job job() throws FileNotFoundException, XMLStreamException, URISyntaxException {
    return jobBuilderFactory.get("job")
            .incrementer(new RunIdIncrementer())
            .start(step1()).next(step2())
            .build();
}

@Bean
public Step step1() {
    return stepBuilderFactory.get("step1")
            .tasklet(taskletStep).listener(promotionListener())
            .build();
}

    @Bean
    public ExecutionContextPromotionListener promotionListener() {
            ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener();

            listener.setKeys(new String[] {"someKey" });

            return listener;
    };


@Bean
public Step step2() {
    return stepBuilderFactory.get("step2")
            .<String, String> chunk(1)
            .reader(myItemReader)
            .writer(myItemWriter)
            .build();
}

This is the taskletStep

private static FileSystem fileSystem = FileSystems.getDefault();
private String locationSourceNetwork = "E:\\assss\\wfwef.txt";
private ChunkContext chunkContext;
String homeOfficeDirectoryUnzip =  fileSystem.getPath("HOME").toAbsolutePath().toString();
@Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {

    //String homeOfficeDirectoryUnzip =  fileSystem.getPath("HOME").toAbsolutePath().toString();
    File file = new File(homeOfficeDirectoryUnzip);

    Path sourceDirectory = Paths.get(locationSourceNetwork);
    //Path targetDirectory = Paths.get(homeOfficeDirectoryUnzip);
    Files.copy(sourceDirectory, file.toPath());
    ExecutionContext stepContext = chunkContext.getStepContext().getStepExecution().getExecutionContext();
    stepContext.put("someKey", homeOfficeDirectoryUnzip);
    return RepeatStatus.FINISHED;
}
@BeforeStep
public void saveStepExecution(ChunkContext chunkContext) {
    this.chunkContext = chunkContext;
    /*ExecutionContext stepContext = this.stepExecution.getExecutionContext();
    stepContext.put("someKey", homeOfficeDirectoryUnzip);*/
}

This is the MyItemReader-

private Resource[] resources;

/*@Autowired
ResourcesFactoryBean fileResources;*/

private static String files =  new ClassPathResource("HOME").toString();//FileSystems.getDefault().getPath("HOME").toAbsolutePath().toString();

private Resource[] resources;


@BeforeStep
public void retrieveInterstepData(StepExecution stepExecution) throws IOException {
    JobExecution jobExecution = stepExecution.getJobExecution();
    ExecutionContext jobContext = jobExecution.getExecutionContext();
    ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();
    this.resources = patternResolver.getResources("file:"+jobContext.get("someKey").toString());
}


public MultiResourceItemReader<String> multiResourceItemReader()
        throws FileNotFoundException, XMLStreamException, URISyntaxException {
    MyItemReader resourceItemReader = new MyItemReader();
    resourceItemReader.setResources(resources);
    resourceItemReader.setDelegate(headerXmlReader());
    resourceItemReader.setStrict(false);
    return resourceItemReader;
}

public StaxEventItemReader<String> headerXmlReader()
        throws FileNotFoundException, XMLStreamException, URISyntaxException {
    // TODO Auto-generated method stub

    StaxEventItemReader<String> staxHeaderDataReader = new StaxEventItemReader<String>();
    System.out.println("HIIIIIIIIIII");
    int i = resources.length;
    while (i > 0) {
        staxHeaderDataReader.setResource(resources[i - 1]);
        i--;
    }
    return staxHeaderDataReader;

}

This resource should have the file which is being copied by TaskletStep but the batch is failing during application startup. Note-this is just a dummy poc of the actual scenario.

Stack Trace--
main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=job]] launched with the following parameters: [{run.id=10}]
2018-10-19 20:35:43.959  INFO 3108 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step1]
2018-10-19 20:35:43.998  INFO 3108 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step2]
2018-10-19 20:35:44.009 ERROR 3108 --- [           main] o.s.batch.core.step.AbstractStep         : Encountered an error executing step step2 in job job

java.lang.IllegalArgumentException: Resources must be set
    at org.springframework.util.Assert.notNull(Assert.java:193) ~[spring-core-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.batch.item.file.MultiResourceItemReader.open(MultiResourceItemReader.java:168) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:103) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:310) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:197) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:394) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_181]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_181]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_181]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at com.sun.proxy.$Proxy89.run(Unknown Source) [na:na]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1258) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at com.infotech.batch.SpringBootBatchTaskletApplication.main(SpringBootBatchTaskletApplication.java:12) [classes/:na]

2018-10-19 20:35:44.014 ERROR 3108 --- [           main] o.s.batch.core.step.AbstractStep         : Exception while closing step execution resources in step step2 in job job

java.lang.NullPointerException: null
    at org.springframework.batch.item.file.MultiResourceItemReader.close(MultiResourceItemReader.java:155) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.item.support.CompositeItemStream.close(CompositeItemStream.java:89) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep.close(TaskletStep.java:305) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:271) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:394) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_181]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_181]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_181]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at com.sun.proxy.$Proxy89.run(Unknown Source) [na:na]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1258) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at com.infotech.batch.SpringBootBatchTaskletApplication.main(SpringBootBatchTaskletApplication.java:12) [classes/:na]

2018-10-19 20:35:44.019  INFO 3108 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=job]] completed with the following parameters: [{run.id=10}] and the following status: [FAILED]

What does _FileSystems.getDefault()."folder_name"_ mean? Please update the question with the step 1 code along with the stack trace. — Andrew S, Oct 18 '18 at 19:31
Also if it can be helped that how the stax event reader should be assigned the resource one by one from the array of resources. — sourav bhardwaj, Oct 18 '18 at 20:13
The step names are confusing: `step2()` refers to "step1"? Can you make those consistent and update which step is actually failing? — Andrew S, Oct 18 '18 at 20:18
Done- step 1 is the tasklet step where as step 2 is the step where i need to read and write. Ignore process and third taskler(as mentioned in the question as they should not be of any problem once this gets resolved). The job is failing in "Encountered an error executing step step2 " — sourav bhardwaj, Oct 18 '18 at 20:31
I tried using ExecutionContextPromotionListener in step one and tried to get the value in 2nd step, i am getting the value in step 2 but don’t know why it is giving exception when going for resource open and close — sourav bhardwaj, Oct 19 '18 at 20:18

Mahmoud Ben Hassine · Answer 1 · 2018-10-19T10:20:10.607

0

You don't need to set the resource manually on the delegate reader here:

private StaxEventItemReader<String> headerXmlReader()
    throws FileNotFoundException, XMLStreamException, URISyntaxException {
   // TODO Auto-generated method stub

   StaxEventItemReader<String> staxHeaderDataReader = new StaxEventItemReader<String>();

   int i = resources.length;
   while (i > 0) {
       staxHeaderDataReader.setResource(resources[i - 1]);
       i--;
   }
   return staxHeaderDataReader;

}

The MultiResourceItemReader will do it automatically while reading the resources. Basically, what you need to do is to configure everything on the delegate (root tag name, unmarshaller, etc) except the resource. The resource will be set dynamically by the MultiResourceItemReader on the delegate.

You can find an example of how to configure a MultiResourceItemReader with a delegate StaxEventItemReader here.

pass value of resource from step 1 to the next step

step1 should dynamically pass the information about the created resource to step2 via the step execution context. For more details about how to do this, see:

Similar SO question: How can we share data between the different steps of a Job in Spring Batch?
"Passing Data to Future Steps" section in the documentation: https://docs.spring.io/spring-batch/4.0.x/reference/html/common-patterns.html#passingDataToFutureSteps.

Hope this helps.

edited Oct 19 '18 at 10:20

answered Oct 19 '18 at 08:32

Mahmoud Ben Hassine

28,519
3
32
50

Thanks for that but the problem is that the resource is created during runtime by tasklet (step 1)and that resource should be used by multiresource reader(in step 2). I cannot provide any hard coded values to the resource as in production there will be no location as “D:/example/example “ , the resource will be having a location created by filesystem.getdefault() (see tasklet 1 code for details). So how to assign this created location to the multi resource reader as during initialisation resource has not been created and so it is null and step 2 is failing when trying to open and close resourc – sourav bhardwaj Oct 19 '18 at 09:32
In this case, step1 should dynamically pass the info about the created resource to step2. I edited the answer with links to how to do that. Hope it helps. – Mahmoud Ben Hassine Oct 19 '18 at 10:20
Thanks, i have tried something similar like this but it did not worked.I think that the batch is failing during the initialisation phase when the application is going up i.e. before the step 1 is running. When spring is going for instantiating the beans (when the server is going up)at that time only it is failing as resource is null(correct me if i am wrong) I will apply the following examples and attach the stack trace if any exception is there. Also i was thinking if I can go for some lazy instantiation of the beans for step 2 so that it gets created after step 1 has executed. – sourav bhardwaj Oct 19 '18 at 11:49
Tried the same but still no luck.. https://drive.google.com/file/d/1U9UZ4m9_ZkN00N_IXK-7gCGxcAFV4GEX/view?usp=sharing for the error – sourav bhardwaj Oct 19 '18 at 13:59

Spring batch pass value of resource from step 1 to the next step

1 Answers1