1

I have a batch and I'm using spring data JPA to query and then update the record. I search the records of a table (RepositoryItemReader) with the condition (code is null). I have a processing logic (ProcessorBatch) that in the end, I update the code value. After that, I update the records (RepositoryItemWriter).

But this search I do every 10 out of 10. The problem happens when I update the code field (which is used in the search condition), with that the paging fails and skips 10 records. Then update 10 records and then skip 10 records, until the end.

In the final log, READ = 5000 WRITE = 5000 SKIP = 0 appears. But there are 10000 records in the database

Every time he comes back to query records, he executes that query before select count (generatedAlias0) from Example as generatedAlias0 where generatedAlias0.code is null

And with that, the total number of records changes, as the code is no longer null.

I did a test if in processing I update another field, other than the code field, it works normally.

Does anyone have any suggestions for me to continue looking for records with pagination so as not to hinder performance and updating normally?

I have the following codes:

BatchConfiguration

    @Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Bean
    public Step exampleStep(ExampleRepository exampleRepository, ProcessorBatch processorBatch, StepBuilderFactory steps)  {
        RepositoryItemReader<Example> reader = getReader(exampleRepository);
        RepositoryItemWriter<Example> writer = getWriter(exampleRepository);

        return steps.get("exampleStep")
                .<Example, Example> chunk(10)
                .reader(reader)
                .listener(new StepListener())
                .processor(processorBatch)
                .writer(writer)
                .faultTolerant()
                .skipPolicy(new CustomSkipPolicy())
                .build();
    }


    private RepositoryItemReader<Example> getReader(ExampleRepository exampleRepository) {
        RepositoryItemReader<Example> reader = new RepositoryItemReader<>();
        reader.setRepository(exampleRepository);
        reader.setMethodName("findByCodeIsNull");
        reader.setSort(Collections.singletonMap("code", Sort.Direction.ASC));
        reader.setPageSize(10);
        return reader;
    }


    private RepositoryItemWriter<Example> getWriter(ExampleRepository exampleRepository) {
        RepositoryItemWriter<Example> writer = new RepositoryItemWriter<>();
        writer.setRepository(exampleRepository);
        writer.setMethodName("save");
        return writer;
    }

 
    @Bean
    public Job job(JobBuilderFactory jobBuilderFactory, ExampleRepository exampleRepository, ProcessorBatch processorBatch, JobBuilderFactory jobs, StepBuilderFactory steps) {
        BatchConfiguration batchConfiguration = new BatchConfiguration();
        return jobs.get("job")
                .incrementer(new RunIdIncrementer())
                .start(batchConfiguration.exampleStep(exampleRepository, processorBatch, steps))
                .build();
    }
    
}   

ProcessorBatch

public Example process(Example example) throws StepSkipException {
        example.setCode("code");
        return example;
    }

MovimentoRepository

@Repository
public interface ExampleRepository extends JpaRepository<Example, Integer> {
    
    Page<Movimento> findByCodeIsNull(Pageable pageable);

}
Deepak Kumar
  • 1,246
  • 14
  • 38
user1748379
  • 53
  • 1
  • 9
  • 1
    Does this answer your question? [Spring batch jpaPagingItemReader why some rows are not read?](https://stackoverflow.com/questions/26509971/spring-batch-jpapagingitemreader-why-some-rows-are-not-read) – Mahmoud Ben Hassine Aug 13 '20 at 09:37
  • @user1748379 Did you find a solution to this using RepositoryItemReader – Sunit Chatterjee Apr 15 '21 at 07:21
  • Yes, create a class by example CustomRepositoryItemReader and extends RepositoryItemReader public class CustomRepositoryItemReader extends RepositoryItemReader { } after that, change the method doPageRead() Pageable pageRequest = PageRequest.of(0, pageSize, sort); put this CustomRepositoryItemReader on the step – user1748379 Apr 16 '21 at 18:24
  • @user1748379 how did you override the doRead method. Can you please explain? How did you manage private instance variables and private methods? – bluelurker Sep 06 '21 at 11:02
  • I copy this class https://github.com/spring-projects/spring-batch/blob/main/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/data/RepositoryItemReader.java and modify the method doPageRead, what i called CustomRepositoryItemReader. in step reader I put this class. – user1748379 Sep 08 '21 at 02:15
  • Ugh isn't there anything more convenient? – html_programmer Oct 04 '21 at 19:46

0 Answers0