Out of memory error while read a file to byte array using apache io utils

Question

I have created a class to generate checksum for the files using apache file utils and bouncy castle to generate checksum. I have read the given file using apache's - IOUtils and I have used bouncy castle's - bcprov to generate checksum.

The code which I used as below :

byte[] digest = null;
String result = "";

SHA1.Digest sha1 = new SHA1.Digest();
sha1.update(IOUtils.toByteArray(new FileInputStream(file)));
digest = sha1.digest();

for (int i = 0; i < digest.length; i++)
{
     result += Integer.toString((digest[i] & 0xff) + 0x100, 16).substring(1);
}

When I try to read a large size of file (i.e about 80MB file) I get out of memory space error as shown below

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteAr rayOutputStream.java:322)
        at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:463)
        at com.test.hashgenerator.HashGenerator.getSHA1CheckSum(HashGe nerator.java:230)
        at com.test.hashgenerator.HashGenerator.main(HashGenerator.jav a:319)

Is I need to enhance my code? Any help would be appreciated.

how much heap space are you giving your JVM? regardless, 80 meg is a huge amount of memory to try to allocate as a byte array. i suggest you push it through the digest in smaller chunks — slipperyseal, Dec 02 '15 at 06:12
check out this: http://stackoverflow.com/questions/6293713/java-how-to-create-sha-1-for-a-file — slipperyseal, Dec 02 '15 at 06:15
Read and update in chunks of small buffers and increase Xmx size — Ravindra babu, Dec 02 '15 at 09:08

score 1 · Answer 1 · edited May 23 '17 at 12:22

Your code crash exactly because of IOUtils.toByteArray(new FileInputStream(file)), SHA doesnt matter.

For more informations, see this for example: Most Robust way of reading a file or stream using Java (To prevent DoS attacks)

This code works fine on a 200 MB file, thanks to @SlipperySeal and this: Java: How to create SHA-1 for a file?

public byte[] createSha1(File file) throws Exception  {
MessageDigest digest = MessageDigest.getInstance("SHA-1");
InputStream fis = new FileInputStream(file);
int n = 0;
byte[] buffer = new byte[8192];
while (n != -1) {
    n = fis.read(buffer);
    if (n > 0) {
        digest.update(buffer, 0, n);
    }
}
return digest.digest();
}

Out of memory error while read a file to byte array using apache io utils

1 Answers1