I have a batch and I'm using spring data JPA to query and then update the record. I search the records of a table (RepositoryItemReader) with the condition (code is null). I have a processing logic (ProcessorBatch) that in the end, I update the code value. After that, I update the records (RepositoryItemWriter).
But this search I do every 10 out of 10. The problem happens when I update the code field (which is used in the search condition), with that the paging fails and skips 10 records. Then update 10 records and then skip 10 records, until the end.
In the final log, READ = 5000 WRITE = 5000 SKIP = 0
appears. But there are 10000 records in the database
Every time he comes back to query records, he executes that query before
select count (generatedAlias0) from Example as generatedAlias0 where generatedAlias0.code is null
And with that, the total number of records changes, as the code is no longer null.
I did a test if in processing I update another field, other than the code field, it works normally.
Does anyone have any suggestions for me to continue looking for records with pagination so as not to hinder performance and updating normally?
I have the following codes:
BatchConfiguration
@Configuration
@EnableBatchProcessing
public class BatchConfiguration {
@Bean
public Step exampleStep(ExampleRepository exampleRepository, ProcessorBatch processorBatch, StepBuilderFactory steps) {
RepositoryItemReader<Example> reader = getReader(exampleRepository);
RepositoryItemWriter<Example> writer = getWriter(exampleRepository);
return steps.get("exampleStep")
.<Example, Example> chunk(10)
.reader(reader)
.listener(new StepListener())
.processor(processorBatch)
.writer(writer)
.faultTolerant()
.skipPolicy(new CustomSkipPolicy())
.build();
}
private RepositoryItemReader<Example> getReader(ExampleRepository exampleRepository) {
RepositoryItemReader<Example> reader = new RepositoryItemReader<>();
reader.setRepository(exampleRepository);
reader.setMethodName("findByCodeIsNull");
reader.setSort(Collections.singletonMap("code", Sort.Direction.ASC));
reader.setPageSize(10);
return reader;
}
private RepositoryItemWriter<Example> getWriter(ExampleRepository exampleRepository) {
RepositoryItemWriter<Example> writer = new RepositoryItemWriter<>();
writer.setRepository(exampleRepository);
writer.setMethodName("save");
return writer;
}
@Bean
public Job job(JobBuilderFactory jobBuilderFactory, ExampleRepository exampleRepository, ProcessorBatch processorBatch, JobBuilderFactory jobs, StepBuilderFactory steps) {
BatchConfiguration batchConfiguration = new BatchConfiguration();
return jobs.get("job")
.incrementer(new RunIdIncrementer())
.start(batchConfiguration.exampleStep(exampleRepository, processorBatch, steps))
.build();
}
}
ProcessorBatch
public Example process(Example example) throws StepSkipException {
example.setCode("code");
return example;
}
MovimentoRepository
@Repository
public interface ExampleRepository extends JpaRepository<Example, Integer> {
Page<Movimento> findByCodeIsNull(Pageable pageable);
}