Store CSV file data in a list/buffer

Question

I'm reading a CSV file with 9 million lines, and I need to turn each line into an object and store it in a list or buffer to later write to a random access file.

When I try to add investor objects to the list, it gives the following error when it reaches 6,462,547 million lines: error Java heap space

How can I get around this error? And how could I replace List with some Buffer method?

Link to the file I'm reading

My reading function (NOTE: The TextFile class can be replaced by BufferedReader):

public List<Object> lerDadosCSV(String arquivoCSV, JProgressBar progressBar, JTextField textField, int tipo) {
    long indice = 0;
    numeroTotalLinhas = numeroTotalLinhas(arquivoCSV) * 2;
    try (TextFile textFile = new TextFile(arquivoCSV)) {
        DecimalFormat decimalFormat = new DecimalFormat("#,###");
        String linha;
        List<Object> records = new ArrayList<>();
        while ((linha = textFile.readLine()) != null) {
            if (indice != 0) {
                records.add(tipo == 0 ? montaEstoque(linha.split(";")) : montaInvestidor(linha.split(";")));
            }
            textField.setText(decimalFormat.format(indice));
            progressBar.setValue((int) (indice * 100 / numeroTotalLinhas));
            progressBar.setString((int) (indice * 100 / numeroTotalLinhas) + "%");
            indice++;
        }
        return records;
    } catch (IOException e) {
        e.printStackTrace();
        return null;
    }
}

Function montaInvestidor:

    public static Investidor montaInvestidor(String[] splitLinha) {
        try {
            boolean operou = (splitLinha[10].contains("s|S")) ? true : false,
                    situacao = (splitLinha[9].contains("a|A")) ? true : false;
            Investidor investidor = new Investidor(Integer.parseInt(splitLinha[0]), formataData.parse(splitLinha[1]),
                    splitLinha[2].trim(), splitLinha[3].trim(), splitLinha[4].trim(), splitLinha[6].trim(),
                    splitLinha[7].trim(), splitLinha[8].trim(), Integer.parseInt(splitLinha[5]), situacao, operou);
            return investidor;
        } catch (NumberFormatException | ParseException e) {
            e.printStackTrace();
            return null;
        }
    }

Does this answer your question? [How to deal with "java.lang.OutOfMemoryError: Java heap space" error?](https://stackoverflow.com/questions/37335/how-to-deal-with-java-lang-outofmemoryerror-java-heap-space-error) — Tim Moore, Aug 24 '21 at 03:16
Consider writing it immediately to file so you don't need to keep it in memory. — Mark Rotteveel, Aug 26 '21 at 14:12

score 0 · Answer 1 · answered Aug 24 '21 at 07:05

Long story short... Heap is always finite and there is a maximum for each platform. In Windows, it is around 2GB. So the default value is pretty small.

How to get around it?

-Xmx

This sets the maximum heap space allocated to this specific program. It is often not recommended, but when you have to use that memory, I personally think it's ok.

So you could either use this with your jar or with your IDE by adding it to the VM Arguments.

An example would be:

java -jar -Xmx4g myJar.jar
4g = 4GB

Or you could use mb and so on.

Store CSV file data in a list/buffer

1 Answers1