Spring Batch - Item Reader and ItemProcessor with a list

Question

I am using a spring batch application which reads a Flat file and returns an object . All i wanted to do is to make the FlatFileItemReader to return a list and pass it to processor and so that it treats each list as one item . Please see the snippet below

@Bean public FlatFileItemReader <List<T>> reader() throws Exception {
    //reader.read()
    }

 @Bean
    public ItemProcessor <List<T>, V> getTargetValueProcessor() {
        return new ItemProcessor <List<T>, V>() {
            @Override
            public V process(List<T> t) throws Exception {
                //processing logic
            }
        };   }

But my Item processor treats each item in the list as a single input to the processor and the processor is called the number of times as the list size. If the list size returned by the reader is 3 , the processor is called three times. Any thoughts on how to handle list inputs in ItemProcessor ??

TIA

Any business logic behind why you want Processor process a list? Because with Spring Batch they have commit-interval and parallel processing, single item or list item for processor actually doesn't matter technically in spring batch — Nghia Do, Jul 14 '18 at 18:12
@NghiaDo My processor will be calling a stored procedure and the input of the stored proc will be an array/collection , so i am doing a chunk based processing where the reader will return a list and the processor will process each chunk as a list and send it to stored proc. — Anu Raman, Jul 15 '18 at 10:17
You can read/process a single item and pass this item to your SP using `Collections.singletonList` and let SB manage chunk lifecycle — Luca Basso Ricci, Jul 15 '18 at 14:37

score 1 · Answer 1 · answered Jul 16 '18 at 12:40

1

You can find an example of an item reader that returns a List of objects as a single item here: https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#multiline

Here is the implementation: https://github.com/spring-projects/spring-batch/blob/master/spring-batch-samples/src/main/java/org/springframework/batch/sample/domain/multiline/AggregateItemReader.java#L55

answered Jul 16 '18 at 12:40

Mahmoud Ben Hassine

28,519
3
32
50

Hi ,I got a chance to look into this code earlier, and what i dont understand is the AggregateItem class. I am unable to get that dependency into my project . Is it something that i have to create on my own or override ? – Anu Raman Jul 17 '18 at 09:11
1

The `AggregateItem` is a wrapper type for an item. In the example, it encapsulates a header, some lines and a footer. You don't have to import it in your project. The example shows an item reader that return a List of records as a single item. Feel free to adapt it to your requirements. – Mahmoud Ben Hassine Jul 17 '18 at 09:40

score 0 · Answer 2 · answered Apr 24 '19 at 14:07

You can write a Generic Class like below which can be used to Read Multiple Lines at once and pass those as List<T> to Processor, Hope it helps :)

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemStreamException;
import org.springframework.batch.item.NonTransientResourceException;
import org.springframework.batch.item.ParseException;
import org.springframework.batch.item.UnexpectedInputException;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.ResourceAwareItemReaderItemStream;
import org.springframework.core.io.Resource;

public class FlatFileItemGroupReader<T> implements ResourceAwareItemReaderItemStream<List<T>> {

private int groupSize = 100;

private static final Logger LOG = LogManager.getLogger(FlatFileItemGroupReader.class);

private FlatFileItemReader<T> fileReader;

@Override
public synchronized List<T> read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
    List<T> records = new ArrayList<T>();

    while(records.size() < groupSize) {
        T record = fileReader.read();
        if (Objects.isNull(record)) {
            break;
        }
        records.add(record);
    }

    if (records.isEmpty()) {
        return null;
    }

    return records;
}

@Override
public synchronized void open(ExecutionContext executionContext) throws ItemStreamException {
    fileReader.open(executionContext);
}

@Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
}

@Override
public synchronized void close() throws ItemStreamException {
    fileReader.close();
}

public FlatFileItemReader<T> getFileReader() {
    return fileReader;
}

public void setFileReader(FlatFileItemReader<T> fileReader) {
    this.fileReader = fileReader;
}

public int getGroupSize() {
    return groupSize;
}

public void setGroupSize(int groupSize) {
    this.groupSize = groupSize;
}

@Override
public void setResource(Resource resource) {
    this.fileReader.setResource(resource);

}
}

I had the similar implementation but the step builder throws an error -- The method reader(ItemReader extends T>) in the type SimpleStepBuilder is not applicable for the arguments (FlatFileItemGroupReader) — Arpit, Aug 12 '21 at 04:49

Spring Batch - Item Reader and ItemProcessor with a list

2 Answers2

Linked