java.lang.OutOfMemoryError while reading file to byte[] array

Question

Is there a cleaner and faster way to do this:

BufferedReader inputReader = new BufferedReader(new InputStreamReader(context.openFileInput("data.txt")));
String inputString;
StringBuilder stringBuffer = new StringBuilder();
while ((inputString = inputReader.readLine()) != null) {
    stringBuffer.append(inputString + "\n");
}
text = stringBuffer.toString();
byte[] data = text.getBytes();

Basically I'm trying to convert a file into byte[], except if the file is large enough then I run into an outofmemory error. I've been looking around SO for a solution, I tried to do this here, and it didn't work. Any help would be appreciated.

Answer 1

Few suggestions:

1. You don't need to create string builder. You can directly read bytes from the file.
2. If you read multiple files, check for those byte[] arrays remaining in memory even when not required.
3. Lastly increase the maximum memory for your java process using -Xmx option.

Answer 2

As we know the size of this file, somewhat half of the memory can be saved by allocating the byte array of the given size directly rather than expanding it:

byte [] data = new byte[ (int) file.length() ];
FileInputStream fin = new FileInputStream(file);
int n = 0;
while ( (n = fin.read(data, n, data.length() - n) ) > 0);

This will avoid allocating unnecessary additional structures. The byte array is only allocated once and has the correct size from beginning. The while loop ensures all data are loaded ( read(byte[], offset, length) may read only part of file but returns the number of bytes read).

Clarification: When the StringBuilder runs out, it allocates a new buffer that is the two times larger than the initial buffer. At this moment, we are using about twice the amount of memory that would be minimally required. In the most degenerate case (one last byte does not fit into some already big buffer), near three times the minimal amount of RAM may be required.

Answer 3

If you haven't enough memory to store there whole file, you can try rethink your algorithm to process file data while reading it, without constructing large byte[] array data.

If you have already tried increase java memory by playing with -Xmx parameter, then there isn't any solution, which will allow you store data in memory, which can not be located there due to its large size.

Answer 4

You are copying bytes into char (which use twice the space) and back into bytes again.

InputStream in = context.openFileInput("data.txt");
ByteArrayOutputStream bais = new ByteArrayOutputStream();
byte[] bytes = new byte[8192];
for(int len; (lne = in.read(bytes) > 0;)
   bais.write(bytes, 0, len);
in.close();
return bais.toByteArray();

This will half your memory requirement but it can still mean you run out of memory. In this case you have to either

• increase your maximum heap size
• process the file progressively instead of all at once
• use memory mapped files which allows you to "load" a file without using much heap.

Answer 5

This is similar to File to byte[] in Java

You're currently reading in bytes, converting them to characters, and then trying to turn them back into bytes. From the InputStreamReader class in the Java API:

An InputStreamReader is a bridge from byte streams to character streams: It reads bytes and decodes them into characters..

It would be way more efficient to just read in bytes.

One way would be to use a ByteArrayInputStream directly on context.openFileInput(), or the Jakarta Commons IOUtils.toByteArray(InputStream), or if you're using JDK7 you can use Files.readAllBytes(Path).

Answer 6

The 'cleaner and faster way' is not to do it at all. It doesn't scale. Process the file a piece at a time.

Answer 7

This solution will test the free memory before loading...

File test = new File("c:/tmp/example.txt");

    long freeMemory = Runtime.getRuntime().freeMemory();
    if(test.length()<freeMemory) {
        byte[] bytes = new byte[(int) test.length()];
        FileChannel fc = new FileInputStream(test).getChannel();
        MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_ONLY, 0, (int) fc.size());

        while(mbb.hasRemaining()) {
            mbb.get(bytes);
        }
        fc.close();
    }