Springbatch dynamic multiple xml File writer

Question

I have to do a batch that :
read some data from DB (each row is an item, this is fine)
then do some process to add some more data (more data is always better ;) )
then here is my problem, I have to write each item in an xml file who's name depends of the item's data.

for exemple I have
ItemA (attr1=toto, attr2=foo, attr3=myNonKeyData...)=>goes in toto_foo.xml
ItemB (attr1=toto, attr2=foo, attr3=myNonKeyData...)=>goes in toto_foo.xml

ItemC (attr1=tata, attr2=foo...)=>goes in tata_foo.xml
...

I can't see how to do this with only one batch run once.
I have too much keys and possible output files to do a classifier.
Maybe using the partitioner could be a good idea even it seems not designed for that.

Why is it a problem to have the writer decide the file to write to? Also. maybe you want to detail how the "more data" from the processor interacts with writer. — Cristian Sevescu, Jan 19 '15 at 12:05

score 5 · Accepted Answer · answered Jan 19 '15 at 21:40

This is what I have understood.

Arbitrary number of files.
Process data from DB in a single pass - read once from DB and write to multiple files.
One or more attributes of the item being written determine the name of the target file to write to.

Create an composite ItemWriter(inspired by org.springframework.batch.item.support.CompositeItemWriter). Here is some pseudocode

public class MyCompositeItemWriter<T> implements ItemStreamWriter<T> {

// The String key is the file-name(toto_foo.xml, tata_foo.xml etc)
//The map will be empty to start with as we do not know how many files will be created. 
private Map<String, ItemWriter<? super T>> delegates;

private boolean ignoreItemStream = false;

public void setIgnoreItemStream(boolean ignoreItemStream) {
    this.ignoreItemStream = ignoreItemStream;
}

@Override
public void write(List<? extends T> items) throws Exception {

    for(T item : items) {
        ItemWriter<? super T> writer = getItemWriterForItem(item);
        // Writing one item ata time might be inefficent. You can optimize this by grouping items by fileName. 
        writer.write(item);     
    }       
}


private getItemWriterForItem(T item) {
    String fileName = getFileNameForItem(item); 
    ItemWriter<? super T> writer = delegates.get(fileName);
    if(writer == null) {
        // There is no writer for the fileName. 
        //create one
        writer = createMyItemWriter(fileName);
        delegates.put(fileName, writer);
    }
    return writer;
}

ItemWriter<? super T> createMyItemWriter(String fileName) {
    // create the writer. Maybe a org.springframework.batch.item.xml.StaxEventItemWriter 
    // set the resource(fielName)
    //open the writer
}




// Identify the name of the target file - toto_foo.xml, tata_foo.xml etc
private String getFileNameForItem(Item item) {
    .....
}

@Override
public void close() throws ItemStreamException {
    for (ItemWriter<? super T> writer : delegates) {
        if (!ignoreItemStream && (writer instanceof ItemStream)) {
            ((ItemStream) writer).close();
        }
    }
}

@Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
    // Do nothing as we do not have any writers to begin with. 
    // Writers will be created as needed. And will be opened after creation. 
}

@Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
    for (ItemWriter<? super T> writer : delegates) {
        if (!ignoreItemStream && (writer instanceof ItemStream)) {
            ((ItemStream) writer).update(executionContext);
        }
    }
}

Nice one ! I've changed the write method because it was writing all in all files. I write element one by one (I keep only one item in the list, the good one), and as I have not thousond of item, it is ok. many thx — bodtx, Feb 02 '15 at 14:39
Another example with a FlatFileItemWriter is available here https://stackoverflow.com/a/16103484/2641426 — DependencyHell, Jun 18 '20 at 09:05

Springbatch dynamic multiple xml File writer

1 Answers1