How to add steps dynamically in a job using spring batch

Question

I am using spring batch where-in I have a use case to configure job with dynamic steps. The number of steps will depend based on the request sent by user. Currently I am using the tasklet methodology to process the step.

I do not want to process the data in chunks.

any way to workaround this, so i can configure job with dynamic steps.

please find the code snippet from JobConfiguration.

@Bean
public Job createBatchJob() {
    return jobFactory.get(JOB_TYPE)
            .preventRestart()
            .start(step1())
            .build();
}

@Bean
public Step step1() {
    return stepFactory.get(STEP_TYPE)
    .tasklet(batchTasklet).build();
}

How can I configure dynamic steps in above configuration?

Possible duplicate of [How to create dynamic steps in Spring Batch](https://stackoverflow.com/questions/54853908/how-to-create-dynamic-steps-in-spring-batch) — Mahmoud Ben Hassine, Nov 04 '19 at 07:58
this is a different case and not an exact duplicate of How to create dynamic steps in Spring Batch. basically we need dynamic steps but to be added lazily to a job , after the job is built. so say i have configured a job, to process some bunch of files. but the user requests a batch and could submit multiple files, now based on no of files user has submitted via an Rest api, i want to add those many number of steps to job created. is this case supported of adding steps to job after its built? or is there a way to create job configuration dynamically. hope this clarifies the question.? — Suchitra, Nov 11 '19 at 06:36
If I understand correctly, the job definition is dynamic and depends on request parameters. So that's is a Spring question rather than a Spring Batch question, you are trying to create beans dynamically (the job/steps beans). You would probably need to inject the bean registry in your controller and update the job/step beans at runtime (See here: https://stackoverflow.com/questions/43051394/how-to-add-bean-instance-at-runtime-in-spring-webapplicationcontext). — Mahmoud Ben Hassine, Nov 12 '19 at 09:06
Conceptually, our job configuration is only one, but it has multiple steps for each execution. Can we use chunk processing for this? A custom itemReader with chunk size as 1. Can each chunk be processed in parallel? — Suchitra, Nov 12 '19 at 10:55

score 0 · Answer 1 · answered Nov 08 '19 at 08:36

this is a different case and not an exact duplicate of How to create dynamic steps in Spring Batch.

basically we need dynamic steps but to be added lazily to a job , after the job is built.

so say i have configured a job, to process some bunch of files. but the user requests a batch and could submit multiple files, now based on no of files user has submitted via an Rest api, i want to add those many number of steps to job created.

is this case supported of adding steps to job after its built?

or is there a way to create job configuration dynamically.

hope this clarifies the question.

reason to do this is we want to track processing of each file seperately and report the status , so step is a natural choice. — Shreyas Deshpande, Nov 08 '19 at 08:42

score 0 · Answer 2 · answered Feb 10 '20 at 06:55

You need to use

SimpleJobBuilder

 // Start your dynamic batch job building with first request sent by the user. 
 SimpleJobBuilder jobBuilder = jobBuilderFactory.get(YOUR_JOB_NAME)
                           .start(new TaskletStep(YOUR_TASKLET_STEP));

You can write your own generic GenericTaskletStep class which takes the user inputs for batch processing. Embed you batch processing logic in that generic class. Then iterate over the inputs and add GenericTaskletStep dynamically.

// Skip the first input that already added in start step above.
for(int i = 1; i < inputs.size(); i++) {
    jobBuilder.next(new GenericTaskletStep(TASKLET_STEP_NAME))
}

// Finally build your job
jobBuilder.build();

How to add steps dynamically in a job using spring batch

2 Answers2