What is the memory efficient way to read a very large csv file of say 3GB in Java?

Question

I have written 2 methods to read the file

 public static void parseCsvFile(String path) throws IOException {
    FileInputStream inputStream = null;
    Scanner sc = null;
    try {
        inputStream = new FileInputStream(path);
        sc = new Scanner(inputStream, "UTF-8");
        while (sc.hasNextLine()) {
            String line = sc.nextLine();
            //logger.info(line);
        }
        // note that Scanner suppresses exceptions
        if (sc.ioException() != null) {
            throw sc.ioException();
        }
    } finally {
        if (inputStream != null) {
            inputStream.close();
        }
        if (sc != null) {
            sc.close();
        }
    }
  }

    public static void parseCsvUsingJavaStream(String path) {
    try (Stream<String> stream = Files.lines(Paths.get(path))) {
         stream.forEach(System.out :: println);
    } catch (IOException ioe) {
        ioe.printStackTrace();
    }
  }

From the first approach what I understand is that the method does not load all the lines from the file into the memory at once, which is memory efficient. I want to achieve the same using lambda expression. My question here is the does my second approach load all the lines into the memory?If yes then how can I make my second approach memory efficient?

Possible duplicate of [Java Read Large Text File With 70million line of text](https://stackoverflow.com/questions/14037404/java-read-large-text-file-with-70million-line-of-text) — Basil Battikhi, Mar 06 '18 at 10:20

score 1 · Answer 1 · answered Mar 06 '18 at 10:22

The answer to your question is in the Files.lines javadoc :

Read all lines from a file as a Stream. Unlike readAllLines, this method does not read all lines into a List, but instead populates lazily as the stream is consumed.

Your second code sample should be roughly as memory-efficient as your first code sample.

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

Using the streams api should result to about the same memory usage as the other approach, unless you parallelize the stream.

From the Javadoc:

Read all lines from a file as a Stream. Unlike readAllLines, this method does not read all lines into a List, but instead populates lazily as the stream is consumed.

Bytes from the file are decoded into characters using the specified charset and the same line terminators as specified by readAllLines are supported.

After this method returns, then any subsequent I/O exception that occurs while reading from the file or when a malformed or unmappable byte sequence is read, is wrapped in an UncheckedIOException that will be thrown from the Stream method that caused the read to take place. In case an IOException is thrown when closing the file, it is also wrapped as an UncheckedIOException.

The returned stream encapsulates a Reader. If timely disposal of file system resources is required, the try-with-resources construct should be used to ensure that the stream's close method is invoked after the stream operations are completed.

What is the memory efficient way to read a very large csv file of say 3GB in Java?

2 Answers2