Spring Batch: How to configure a FlatItemFileReader to read a compressed ND-JSON file?

Question

Input

New line delimited JSON, sample:

{"summersolstice":"21st June, 2019", //complex objects, arrays, etc ... }
{"summersolstice":"21st June, 2018", //complex objects, arrays, etc ... }
{"summersolstice":"21st June, 2017", //complex objects, arrays, etc ... }

Constraint

While I understand the best way is to be using the shiny new JSON item reader devs released last summer (link), it isn't feasible to update the batch version to the latest one yet. That is the only constraint.

Present Approach

As of now, I followed this stack answer, but I don't think having a T for FlatFileItemReader as Map<String, Object> is the best strategy! As of now I just take it in with this code:

public class JsonItemReader extends FlatFileItemReader<Map<String, Object>> {

    public JsonItemReader(File file) {
        Resource resource = new BzipLazyResource(file); //on the fly unzipping
        setResource(resource);
        setLineMapper(new JsonLineMapper());
    }

    public JsonItemReader(String sourceFileName) {
        this(new File(sourceFileName));
    }
}

...and then parse it simply in an ItemProcessor, like:

public class JsonItemProcessor implements ItemProcessor<Map<String, Object>, List<Json>> {

    private ObjectMapper mapper = new ObjectMapper();
    private static final Logger logger = LoggerFactory.getLogger(JsonItemProcessor.class);

    public List<Json> process(Map<String, Object> jsonItem) throws Exception {
        JsonNode jsonNode = mapper.valueToTree(jsonItem);
        return parseJsonItems(jsonNode);
    }

Resources:

• Spring Batch: How to setup a FlatItemFileReader to read from a JSON file?
• How to convert HashMap to JsonNode with Jackson
• New line delimited JSON
• How to read compressed files on the fly
• Batch 4.1.x JsonItemReader

Answer 1

There is no need for such an indirection by extending FlatFileItemReader and creating an additional item processor to convert Map<String, Object> to a List<Json>..

Here is a quick line mapper you can use with the FlatFileItemReader:

import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.batch.item.file.LineMapper;

public class NDJsonLineMapper<T> implements LineMapper<T> {

    private Class<? extends T> targetType;

    private ObjectMapper objectMapper = new ObjectMapper(); // TODO could make this configurable

    public NDJsonLineMapper(Class<? extends T> targetType) {
        this.targetType = targetType;
    }

    @Override
    public T mapLine(String line, int lineNumber) throws Exception {
        return objectMapper.readValue(line, targetType);
    }
}

And its test:

import org.junit.Assert;
import org.junit.jupiter.api.Test;

class NDJsonLineMapperTest {

    @Test
    void testNDJsonMapping() throws Exception {
        // given
        String jsonLine = "{\"id\":1,\"name\":\"foo\"}";
        NDJsonLineMapper<Person> lineMapper = new NDJsonLineMapper<>(Person.class);

        // when
        Person person = lineMapper.mapLine(jsonLine, 1);

        // then
        Assert.assertEquals(1, person.getId());
        Assert.assertEquals("foo", person.getName());

    }

    static class Person {

        private int id;
        private String name;

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }
    }

}

This is an example of option 1.2 in the link you shared. You can use it with the FlatFileItemReader as follows:

@Bean
public FlatFileItemReader<Person> itemReader() {
    return new FlatFileItemReaderBuilder<Person>()
            .name("NDJsonItemReader")
            .resource(new FileSystemResource("file.ndjson"))
            .lineMapper(new NDJsonLineMapper<>(Person.class))
            .build();
}

Hope this helps.