MappingIterator.readAll() generates a list 2 or 3 times bigger than its-self

Question

I am trying to parse a csv file to a pojo. To do this I am using jackson-dataformat-csv. I am testing the scalablity of my program and while time does not seem to be an issue the memory it uses is. I am using just over a million records in a csv to parse into my pojos. The problem is the server that will have this running on will have about 1.5GB ram and so far I have found that it is using almost 2GB at just over a million records. My problem is when I use this:

 MappingIterator<T> objects = csvMapper.readerFor(pojoClass).with(csvSchema).readValues(csvFile);
 List<T> list = objects.readAll();

the list I get back is almost 2GB. When I create a list of a million pojos I use almost no memory. I am creating that list like this:

    List<Pojo> pojos = new ArrayList<>();
    for(int i = 0; i < 1000000; i++){
        Pojo newPojo= getNewPojo();
        pojos.add(newPojo);
    }

I am not sure why the first list I get back is so large when I can create a different list with the same number of objects just a different way of putting them there and have that list so small. Please let me know if I am doing something wrong and how I can fix this issue I am having. Thanks.

score 0 · Answer 1 · edited May 23 '17 at 12:22

I'm not sure about the library you are using, but if you can't find a solution maybe try uniVocity-parsers to read your pojos iteratively?

CsvParserSettings parserSettings = new CsvParserSettings();
//...configure the parser

CsvRoutines routines = new CsvRoutines(parserSettings); 
for (Pojo address : routines.iterate(Pojo.class, csvFile, "UTF-8")) {
    //process your bean
}

Hope this helps, I recently answered a similar question here.

Disclaimer: I'm the author of this library, it's open-source and free (Apache 2.0 license)

MappingIterator.readAll() generates a list 2 or 3 times bigger than its-self

1 Answers1