OutOfMemoryError: Java heap space when trying to read large file

Question

I'm trying to read large file(approximately 516mb), and it has 18 lines of text. I tried to write down the code myself and got an error in the first line of code while trying to read the file:

 try(BufferedReader br = new BufferedReader(new FileReader("test.txt"))) {
        String line;
        while ((line = br.readLine()) != null) {
            String fileContent = line;
        }
 }

Note: File exists and it's size is approximately 516mb. If there is another safer and faster method of reading from please tell me(Even if it will linebreaks). Edit: Here I tried by using Scanner, but it lasts a bit longer and then gives the same error

try(BufferedReader br = new BufferedReader(new FileReader("test.txt"))) {
    Scanner scanner = new Scanner(br);
    while(scanner.hasNext()){
        int index = Integer.parseInt(scanner.next());
        // and here do something with index
    }
}

I even splitted file into 1800 lines, but nothing got fixed

Answer 1

Using BufferedReader already help you to avoid loading the whole file into the memory. So, for further improvement, as you mentioned each number is separated by space, so instead of this:

line = br.readLine();

We can wrap the reader with a scanner,

Scanner scanner = new Scanner(br);

And extract each number in the file using scanner.next(); and store it into an integer array will also help to reduce memory usage:

int val = Integer.parseInt(scanner.next());

This will help you avoiding reading the whole sentence.

And you can also limit your buffer size for BufferedReader

BufferedReader br = new BufferedReader(new FileReader("test.txt") , 8*1024);

More information Does the Scanner class load the entire file into memory at once?

Answer 2

Increase you heap size witn -Xmx.

For your file I would suggest a setting of -Xmx1536m at least as a file size of 516M will increase while loading. Internally Jaava uses 16 bits to represent a character therefore a file with a text of 10 bytes will take approx. 20 bytes as String (except when using UTF-8 with many composed characters).

Answer 3

EDIT It is the same for java heap space, declare variables inside or outside the loop.

Just an advice.

if you can, you shouldn't declare variables inside the loops, because of this, you can fill up the java heap space. In this example, if it were possible, it would be better:

try(BufferedReader br = new BufferedReader(new FileReader("test.txt"))) {
        String line;
        String fileContent;
        while ((line = br.readLine()) != null) {
            fileContent = line;
        }
 } 

Why? Because in each iteration java is reserving new space in heap for the same variable (Java is considering a new diferent variable (you might want this, but probably not)) and if the loop is big enough, the heap can be full.

Answer 4

Java was designed to work with big amount of data which is bigger then available memory. On lover level API file is a stream, possibly endless.

However with chip memory people prefer easy way - read all to the memory and work with memory. Usually it works but not in your case. Increasing memory only hides this issue till you have bigger file. So, it's time to do it right.

I don't know your sorting approach what you use for comparison. If it is good one then it may produce some sortable key or index of each string. You read file once, create map pf such keys, sort them and then create sorted file based on this sorted map. That would be (worst case scenario) in your case 1+18 file readings plus 1 writing.

However if you don't have such key and simply compare strings character by character then you have to have 2 input streams and compare one to another. If one string is not in correct place then you rewrite file in correct order and do it again. Worst case scenario 18*18 readings to compare, 18*2 reading for writing and 18 writings.

That's the consequence for such architecture when you keep your data in huge strings in huge files.

Answer 5

Note: Increasing the heap memory limit to sort a file with 18 lines is just a lazy way to solve a programming problem, this philosophy of always increase the memory instead of solving the real problem is a reason of Java programs bad fame about slowness and such.

My advice, to avoid increasing the memory for such a task is to split the file by line and merge the lines in a way that resembles a MergeSort. This way your program can scale up if the file size grows.

To split the file in several "line sub files", use the read method of the BufferedReader class:

private void splitBigFile() throws IOException {
    // A 10 Mb buffer size is decent enough
    final int BUFFER_SIZE = 1024 * 1024 * 10; 

    try (BufferedReader br = new BufferedReader(new FileReader("test.txt"))) {
        String line;

        int fileIndex = 0;
        FileWriter currentSplitFile = new FileWriter(new File("test_split.txt." + fileIndex));

        char buffer[] = new char[BUFFER_SIZE]; 

        int readed = 0;
        while ((readed = br.read(buffer)) != -1) {
            // Inspect the buffer in search of the new line character
            boolean endLineProcessed = false;
            for (int i = 0; i < readed; i++) {
                if (buffer[i] == '\n') {
                    // This chunk contains the new line character, write this last chunk the current file and create a new one
                    currentSplitFile.write(buffer, 0, i);
                    fileIndex++;
                    currentSplitFile = new FileWriter(new File("test_split.txt." + fileIndex));
                    currentSplitFile.write(buffer, i, readed - i);
                    endLineProcessed = true;
                }
            }

            // If not end of line found, just write the chunk 
            if (!endLineProcessed) {
                currentSplitFile.write(buffer, 0, readed);
            }
        }
    }
}

To merge them, open all the files and keep a separate buffer (a small one, like 2 mb each) for every one of them, read the first chunk of every file and there you'll have enough information to start rearranging the index of the files. Keep reading chunks if some of the files have ties.

Answer 6

It's hard to guess without understanding of a memory profile of your application, your JVM settings and hardware. It could be as simple as just changing JVM memory settings or as hard as going with RandomFileAccess and convert bytes on your own. I'll try a long shot here. The problem can just lie with the fact that you are trying to read very long lines not with the fact the file is large.

If you look at the implementation of BufferedReader.readLine() you'll see something like this (simplified version):

String readLine() {
  StringBuffer sb = new StringBuffer(defaultStringBufferCapacity);  
  while (true) {
    if (endOfLine) return sb.toString();
     fillInternalBufferAndAdvancePointers(defaultCharBufferCapacity);//(*)
     sb.append(internalBuffer); //(**)
  }
}
// defaultStringBufferCapacity = 80, can't be changed 
// defaultCharBufferCapacity = 8*1024, can be altered

(*) Is the most critical line here. It tries to fill internal buffer of limited size 8K and append the char buffer to StringBuffer. 516Mb file with 18 lines means each line will occupy ~28Mb in memory. So it tries to allocate and copy 8K array ~3500 times per line.

(**)Then it tries to put this array into StringBuffer of default capacity 80. This causes extra allocations for StringBuffer to ensure it's internal buffer is large enough to keep the string ~ 25 extra allocations per line if I'm not mistaken.

So basically, I'd recommend to increase size of the internal buffer to 1Mb, just pass extra parameter to the instance of BufferedReader like:

 new BufferedReader(..., 1024*1024);