0

My job is to get data from CSV between two dates. For this firstly I have loaded all data into a list of beans. Then loop through the list and prepared my result.

The problem is in this process I have to loop through a lot of data.

I want a solution where I will load only necessary data from CSV. Ultimately I want to reduce the operations I am doing.

beansExtractedFromCsv = new CsvToBeanBuilder(new FileReader(s3StorageLocation.concat(s3FileName))).withType(CsvExtractTemplate.class).build().parse();

In this line, I am parsing all data from CSV

dataWriteOnCSVOrDB(fromTime, toTime, false, beansExtractedFromCsv);

Here I am passing all data extracted from CSV to my method, where I am looping through a lot of data to calculate my expected result.

There is another option in OpenCsv to read line by line. I haven't tried that.

naimul
  • 69
  • 1
  • 4
  • 1
    csv's can be read the same as txt files, have you tried that? – Stultuske Sep 17 '21 at 05:33
  • No, Ok I am trying this – naimul Sep 17 '21 at 05:35
  • Like @Stultuske said but to elaborate a bit, you can use `Stream`, see this answer https://stackoverflow.com/a/17866514/1745616 Reading and processing it line by line will cut memory use significantly. Only down-side is if your data contains values with line-breaks (using quotes) then this method might work incorrectly. But if you know your data doesn't contain this then it will work fine. – BdR Sep 17 '21 at 09:46

0 Answers0