How to access read raw data in a spring batch job (xml or csv)

Question

I have different import jobs to get data into our system. Since the data is coming from different producers, i sometimes have csv data, or different formats of xml. Each of these import types has its own spring batch job with its own configuration (different readers, different processors but all the same writer).

My task now is, that i also need the "raw data" in my system and not only the converted objects. So for a csv import i want to have access to the raw line that builds up one entity. In XML i want the raw element as string. So the writer should be extended to take my converted object DTO plus an additional string with the raw data that was read.

I just can´t figure out how to access the raw data with spring batch. I tried several ways to get into the processing line with ItemReadListener#beforeRead or afterRead but i cannot access the raw data from the files.

Any ideas for what i can look further? Or tips on how to achive getting the raw data + the converted dto objects?

"*I tried several ways*" then show us – Michael Mar 10 '20 at 09:03 — Michael, Mar 10 '20 at 09:03

score 1 · Answer 1 · answered Mar 10 '20 at 10:04

You do not have to map data to a domain objects, your items can be of type String.

For flat files, you can use the PassThroughLineMapper which will give you the raw line verbatim:

@Bean
public FlatFileItemReader<String> itemReader() {
    return new FlatFileItemReaderBuilder<String>()
            .name("rawDataReader")
            .resource(new FileSystemResource("/absolute/path/to/your/flat/file"))
            .lineMapper(new PassThroughLineMapper())
            .build();
}

For XML files, you can use the same approach with a StaxEventItemReader<String>. However, Spring Batch delegates the unmarshalling process to org.springframework.oxm.Marshaller, so depending on which XML implementation you use, you need to configure the unmarshaller accordingly:

@Bean
public StaxEventItemReader<String> itemReader() {
    Marshaller marshaller = .. // create or inject marshaller
    // configure marshaller to unmarshal raw strings
    return new StaxEventItemReaderBuilder<String>()
            .name("rawXmlDataReader")
            .resource(new FileSystemResource("/absolute/path/to/your/xml/file"))
            .addFragmentRootElements("yourTagName")
            .unmarshaller(marshaller)
            .build();
}

For Jaxb, this might help: JAXB use String as it is

Thank you! We will take a look at it and try to implement it. — jubi, Mar 10 '20 at 10:32

How to access read raw data in a spring batch job (xml or csv)

1 Answers1