Spring Batch skip exception for ItemWriter

Question

I'm trying to use Spring Batch 2.2.5 with Java config. Here is the config that I have:

@Configuration
@EnableBatchProcessing
public class JobConfiguration {
    @Autowired
    private JobBuilderFactory jobBuilder;

    @Autowired
    private StepBuilderFactory stepBuilder;

    @Bean
    @Autowired
    public Job processDocumentsJob() {
        return jobBuilder.get("processDocumentsJob")
                .start(procesingStep())
                .build();
    }

    @Bean
    @Autowired
    public Step procesingStep() {
           CompositeItemProcessor<File, DocumentPackageFileMetadata> compositeProcessor = new CompositeItemProcessor<File, DocumentPackageFileMetadata>();
           compositeProcessor.setDelegates(Lists.newArrayList(
                   documentPackageFileValidationProcessor(),     
                   documentMetadataFileTransformer()
           ));
        return stepBuilder.get("procesingStep")
                .<File, DocumentPackageFileMetadata>chunk(1)
                .reader(documentPackageFileReader())
                .processor(compositeProcessor)
                .writer(documentMetadataFileMigrator())
                .faultTolerant()
                .skip(DocumentImportException.class)
                .skipLimit(10)
                .listener(stepExecutionListener())
                .build();
    }
....


}

With the config above, if the itemwriter (the bean pointed by documentMetadataFileMigrator) throws 'DocumentImportException', then the exception wont be skipped. Spring Batch will actually retry the same input again. i.e. it will use the same input against 'documentPackageFileValidationProcessor'.

But, if I move the logic inside the itemwriter into an itemprocessor:

 @Bean
        @Autowired
        public Step procesingStep() {
               CompositeItemProcessor<File, DocumentPackageFileMetadata> compositeProcessor = new CompositeItemProcessor<File, DocumentPackageFileMetadata>();
               compositeProcessor.setDelegates(Lists.newArrayList(
                       documentPackageFileValidationProcessor(),     
                       documentMetadataFileTransformer(),
                       documentMetadataFileMigratorAsProcessor() // same as itemwriter, but implemented as itemprocessor
               ));
            return stepBuilder.get("procesingStep")
                    .<File, DocumentPackageFileMetadata>chunk(1)
                    .reader(documentPackageFileReader())
                    .processor(compositeProcessor)                    
                    .faultTolerant()
                    .skip(DocumentImportException.class)
                    .skipLimit(10)
                    .listener(stepExecutionListener())
                    .build();
        }

then the exception will be skipped correctly. i.e. Spring Batch will not retry the same item against 'documentPackageFileValidationProcessor'. It will go to the next item to process (the one returned from 'documentPackageFileReader').

Is this a bug on Spring Batch, or is it behaving as expected? If so, can someone point me to the relevant documentation?

Thanks guys, and apology if this is a fundamental question.

Best regards,

Alex

score 18 · Accepted Answer · edited May 23 '17 at 11:54

18

That behavior is correct. The ItemWriter receives a list of items to write. If a skippable exception is thrown, Spring Batch attempts to determine which item actually caused the exception so only that item is skipped. The way this is done is the transaction is rolled back, the commit interval is changed to 1, and each item is then reprocessed and the write is attempted again. This allows only the item with the error to be skipped instead of needing to skip the entire chunk.

This same issue is discussed here (only using XML config): How is the skipping implemented in Spring Batch?

edited May 23 '17 at 11:54

Community

1
1

answered Mar 20 '14 at 13:14

Michael Minella

20,843
4
55
67

Hi Michael. Thanks for the reply. What I am seeing is as follows: let's say there are 3 items, and item 1 will cause write failure. First run of write will stop due to the exception, and of course, the rest of the items won't get written. Spring then tries to write again (as you said) but with all of the items (including item 1). This will cause failure again. And that's it. There's no attempt to write item 2 and 3 – caffeine_inquisitor Mar 20 '14 at 20:18
I just stepped through the app again. I now know why it didnt process the rest of the items. It is because during the retry, one of the itemprocessor throw an exception that is not skippable. Silly me! Thanks for the great pointer, Michael – caffeine_inquisitor Mar 20 '14 at 21:59
2

Actually... come to think about it again - if the chunk size is 1, then it shouldnt need to go back and retry the whole chain again just to figure out which item causes an error, right? I mean... if chunk size is 1, and the configuration is skip the exception, no retry for that exception... then just go ahead and process the next item. – caffeine_inquisitor Mar 20 '14 at 22:25
2

The ItemWriter#write method receives a list of items. Without going through them one at a time, there is no way for us to determine which in the list threw the exception in the writer. All the framework knows is that something in that list caused something to break. – Michael Minella Mar 20 '14 at 22:30
For me, somehow it is behaving a little differently. Say, i have 4 records to be read-processed-write in chunk-size of 10. Record-2 has some problem which throws a skippable exception during "process" state. It skips it correctly, but writes only Record-3 and Record-4. Record-1 which was correctly read & processed, does not get written at all. What can be wrong? – MiKu Nov 24 '14 at 02:59
@MichaelMinella although, it is an old post but wants to know what will happen if we have composite writers. https://stackoverflow.com/questions/62320057/spring-batch-skip-exception-is-skipping-the-remaining-writers – JDev Jun 12 '20 at 00:22

score 3 · Answer 2 · answered Mar 20 '14 at 21:58

At the end, this is what is working for me - if I want to use itemwriter, with no reprocessing of the same item:

 @Bean
    @Autowired
    public Step procesingStep() {
           CompositeItemProcessor<DocumentPackageFileMetadata, DocumentPackageFileMetadata> compositeProcessor = new CompositeItemProcessor<DocumentPackageFileMetadata, DocumentPackageFileMetadata>();
           compositeProcessor.setDelegates(Lists.newArrayList(
                   documentPackageFileValidationProcessor(),
                   documentPackageFileExtractionProcessor(),
                   documentMetadataFileTransformer()
           ));
        return stepBuilder.get("procesingStep")
                .<DocumentPackageFileMetadata, DocumentPackageFileMetadata>chunk(1)
                .reader(documentPackageFileReader())
                .processor(compositeProcessor)
                .writer(documentMetadataFileMigrator())
                .faultTolerant()
                .skip(DocumentImportException.class)
                .noRetry(DocumentImportException.class)
                .noRollback(DocumentImportException.class)
                .skipLimit(10)
                .listener(skipListener())
                .listener(documentPackageReadyForProcessingListener())
                .listener(stepExecutionListener())
                .build();
    }

Note that I have specified 'noRetry' and 'noRollback'.

Not working for me, it still retries the chunk again, even with the noRetry and noRollBack, ending up with 2 error logs for the same record. — Cristiano Fontes, Aug 24 '15 at 14:23
@CristianoFontes I understand it's been a while, but have you found any solution to this? — Deepanshu Rathi, Jan 12 '22 at 09:18

Spring Batch skip exception for ItemWriter

2 Answers2

Linked