FileUtils.readFileToByteArray vs Files.readAllBytes

Question

I was facing below exception when I was making use of Files.readAllBytes from java 7 which then got solved by using FileUtils.readFileToByteArray from apache. I took that decision by going through few answers in stackoverflow. I tried understanding by going through the code to know the main difference which can solve this issue but in vain. I just want to know what difference in these two that solved this exception.

  java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:658)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
    at sun.nio.ch.IOUtil.read(IOUtil.java:196)
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at java.nio.file.Files.read(Files.java:2903)
    at java.nio.file.Files.readAllBytes(Files.java:2953)

You should really use `Files.readAllBytes()`, but then how large are your files? To me it looks like you are using a 32bit JVM and a large max heap size; am I correct? — fge, Apr 02 '15 at 06:47
Yes, even I feel we should go with java.nio. My files sizes vary from 20MB to 30MB. No, I'm using 64bit JVM and my heap size is set to -Xmx512m. — RAMSY, Apr 02 '15 at 07:02
Huuh, now that is strange; there should be plenty of space for so-called "direct" memory then... Increasing your heap size won't help here. Honestly I'm puzzled. — fge, Apr 02 '15 at 07:04
What OS is that? If Linux, can you edit your question and add the output of `ulimit -a` _as the user which is running the program_? — fge, Apr 02 '15 at 07:05
My OS is Windows 7, what about this post especially the last comment. http://stackoverflow.com/questions/11741804/java-file-to-byte-array-fast-one — RAMSY, Apr 02 '15 at 07:11
Well, I know nada about Windows, so I can't help more than that, sorry ;) — fge, Apr 02 '15 at 07:16

kucing_terbang · Accepted Answer · 2015-04-02T14:13:59.807

Well, you can increase the buffer size by using this command. It may solve your error.

-XX:MaxDirectMemorySize=256M

I tried to cursory read the two source code and the differences that i can found is that apache method is not trying to read all the file and store it into the memory but it write to an output first (which is their own implementation of ByteArrayOutputStream) which is to an ArrayList (so you can say that they put use the memory space that you have increased through -Xmx). and the other one is just store all content into a memory space which could be the reason of your error.

FileUtils.readFileToByteArray vs Files.readAllBytes

1 Answers1