Spring Batch : How does ClassifierCompositeItemWriter decide whether to create a new writer or use an existing one?

Question

Given a Spring Batch job that uses chunk oriented processing, I have a requirement to write the same data to different files depending on the value of an attribute. In other words, I need the output file name be decided at runtime depending on the value of an attribute.

I found this answer on SO and it looks like I can use a ClassifierCompositeItemWriter to dynamically create writers at runtime.

What I want to understand is that whether the ClassifierCompositeItemWriter will create a new ItemWriter instance for every single write? If not, how does the ClassifierCompositeItemWriter know what attribute value from the domain object to use for caching the ItemWriter instances so that they can be reused? (rather than creating a new ItemWriter per object..)

Do I need to implement the caching of wrtiers myself or is this handled by the ClassfierCompositeItemWriter automatically?

Assuming that the cahcing of writers needs to implemented manually, I came up with the following implementation. However, I would assume this is something ClassfierCompositeItemWriter would be doing itself?

public abstract class AbstractFileWriterClassifier<T> implements Classifier<T, ItemWriter<T>> {

private static final long serialVersionUID = 906600779221731593L;

private transient LineAggregator<T> lineAggregator;
private transient FlatFileHeaderCallback headerCallback;
private transient String baseFilePath;
private transient boolean appendAllowed;

private transient Map<String, ItemWriter<T>> cachedItemWriters;

public AbstractFileWriterClassifier() {
    cachedItemWriters = new ConcurrentHashMap<>(5);
}

@Override
public final ItemWriter<T> classify(T item) {
    final String dynamicFilePath = getDynamicFilePath(item);

    // Get existing writer or create a new one if it doesn't exist for the key
    return cachedItemWriters.computeIfAbsent(dynamicFilePath, dynamicPath -> {
        String fileName = baseFilePath + dynamicPath;

        FlatFileItemWriter<T> itemWriter = new FlatFileItemWriter<>();
        itemWriter.setResource(new FileSystemResource(fileName));
        itemWriter.setAppendAllowed(appendAllowed);
        itemWriter.setLineAggregator(lineAggregator);
        itemWriter.setHeaderCallback(headerCallback);
        itemWriter.open(new ExecutionContext());

        return itemWriter;
    });

}

/**
 * Derives the dynamic part of the file path using the item
 * 
 * @param item
 * @return
 */
protected abstract String getDynamicFilePath(T item);

}

`depending on the value of an attribute` where does this attribute come from? Is it one of the fields of the item? — Mahmoud Ben Hassine, Nov 18 '19 at 15:01
Yes that's right. I would have assumed `ClassifierCompositeItemWriter` would cache writers so that a new writer is not created per item but it doesn't look like that's the case. Ideally, that is how it should have been. Also, it seems to iterate over the items twice. Once to group them by writer and then to write them in groups. — Ping, Nov 19 '19 at 02:22
See example here: https://stackoverflow.com/a/53388876/5019386. Hope it helps. — Mahmoud Ben Hassine, Nov 19 '19 at 07:50
@MahmoudBenHassine Thanks. I did come across the linked answer. My primary question was whether `ClassifiercompositeItemWriter` caches the writer instances internally. I went through the *Spring Batch* code and found that it doesn't cache writer instances. The `Classifier` can be implemented such that it caches the writer instances. My question is answered. Thank you :) — Ping, Nov 19 '19 at 13:42

Spring Batch : How does ClassifierCompositeItemWriter decide whether to create a new writer or use an existing one?

0 Answers0