My Java program which reads a large text file is running out of memory, can anyone help explain why?

Question

I have a large text file with 20 million lines of text. When I read the file using the following program, it works just fine, and in fact I can read much larger files with no memory problems.

public static void main(String[] args) throws IOException {
    File tempFile = new File("temp.dat");
    String tempLine = null;
    BufferedReader br = null;
    int lineCount = 0;
    try {
        br = new BufferedReader(new FileReader(tempFile));
        while ((tempLine = br.readLine()) != null) {
            lineCount += 1;
        }
    } catch (Exception e) {
        System.out.println("br error: " +e.getMessage());
    } finally {
        br.close();
        System.out.println(lineCount + " lines read from file");
    }
}

However if I need to append some records to this file before reading it, the BufferedReader consumes a huge amount of memory (I have just used Windows task manager to monitor this, not very scientific I know but it demonstrates the problem). The amended program is below, which is the same as the first one, except I am appending a single record to the file first.

public static void main(String[] args) throws IOException {
    File tempFile = new File("temp.dat");
    PrintWriter pw = null;
    try {
        pw = new PrintWriter(new BufferedWriter(new FileWriter(tempFile, true)));
        pw.println(" ");
    } catch (Exception e) {
        System.out.println("pw error: " + e.getMessage());
    } finally {
        pw.close();
    }

    String tempLine = null;
    BufferedReader br = null;
    int lineCount = 0;
    try {
        br = new BufferedReader(new FileReader(tempFile));
        while ((tempLine = br.readLine()) != null) {
            lineCount += 1;
        }
    } catch (Exception e) {
        System.out.println("br error: " +e.getMessage());
    } finally {
        br.close();
        System.out.println(lineCount + " lines read from file");
    }
}

A screenshot of Windows task manager, where the large bump in the line shows the memory consumption when I run the second version of the program.

So I was able to read this file without running out of memory. But I have much larger files with more than 50 million records, which encounter an out of memory exception when I run this program against them? Can someone explain why the first version of the program works fine on files of any size, but the second program behaves so differently and ends in failure? I am running on Windows 7 with:

java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) Client VM (build 23.1-b03, mixed mode, sharing)

Answer 1

you can start a Java-VM with VM-Options

-XX:+HeapDumpOnOutOfMemoryError

this will write a heap dump to a file, which can be analysed for finding leak suspects

Use a '+' to add an option and a '-' to remove an option.

If you are using Eclipse the Java Memory Analyzer Plugin MAT to get Heap-Dumps from running VMs with some nice analyses for Leak Suspects etc.

Answer 2

Each time you execute the java following Java routine, you are creating a brand new object:

tempLine = br.readLine()

I believe each time you call readLine() it is probably creating a new String object which is left on the heap each time the re-assignment is called to assign the value to tempLine.

Therefore, since GC isn't constantly being called thousands of objects can be left on the heap within seconds.

Some people say its a bad idea to call System.gc() every 1000 lines or so but I would be curious if that fixes your issue. Also, you could run this command after each line to basically mark each object as garbage collectable:

tempLine=null;

Answer 3

     pw = new PrintWriter(new BufferedWriter(new FileWriter(tempFile, true)));

did you try not using a BufferedWriter? If your appending a few lines to the end maybe you don't need a buffer? If you do, consider using a byte array (collections or String builder). Finally did you try the same in java 1.6_32? Might be a bug in the new version of one of the Writers.

Can you print the free memory after before and after pw.close(); ?

System.out.println("before wr close :"  + Runtime.getRuntime().freeMemory());

and similar after close and after reader close

Answer 4

It could be because you may not be having linefeed/carriage return in your file at all. In this case, readLine() tries to create just one single string out of your file which is probably running out of mememory.

Java doc of readLine():

Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed.

Answer 5

Have you tried:

A) creating a new File instance to use for the reading, but pointing to the same file. and B) reading an entirely different file in the second part.

I'm wondering if either, the File object is still somehow attached to the PrintWriter or if the OS is doing something funny with the file handles. Those tests should show you where to focus.

This doesn't look to be a problem with the code, and your logic for thinking it shouldn't break seems sound, so it's got to be some underlying functionality.

Answer 6

you'll need to start java with a bigger heap. Try -Xmx1024m as a parameter on the java command.

Basically your going to need more memory than the size of the file.