java.lang.OutOfMemoryError: Java heap space on writing to outputstream

Question

I'm trying to encrypt a file by passing the file stream, the resultant stream and the cipher object which has the information regarding the type of encryption. Here's what I've tried :

private void processFile(Cipher cipher,InputStream inputStream, OutputStream outputStream){
   byte[] tempInputBuffer = new byte[1024];
   int len;
   try {
        while ((len = inputStream.read(tempInputBuffer)) != -1) {
            byte[] tempOutputBuffer = cipher.update(tempInputBuffer, 0, len);
            if ( tempOutputBuffer != null ) outputStream.write(tempOutputBuffer);
        }
        byte[] obuf = cipher.doFinal();
        if ( obuf != null ) outputStream.write(obuf);
        }catch (IOException | IllegalBlockSizeException | BadPaddingException e) {
            e.printStackTrace();    
        }catch (Exception e) {
            e.printStackTrace();        
        }
    }

This is the function from where I make the call to processFile

public ByteArrayOutputStream encryptFile( InputStream fileStream, PublicKey publicKey ) throws EmprisException {

        ByteArrayOutputStream encryptedFileStream = null;
        KeyGenerator keyGenerator;
        try {

            //generate AES key
            keyGenerator = KeyGenerator.getInstance(FileUploadDownloadConstants.AES_ALGORITHM);
            keyGenerator.init(128);
            SecretKey secretKey = keyGenerator.generateKey();
            byte[] initializationVector = new byte[16];
            SecureRandom srandom = new SecureRandom();
            srandom.nextBytes(initializationVector);
            IvParameterSpec ivSpec = new IvParameterSpec(initializationVector);

            //encrypting the aes key using rsa public key and adding it to a file
            encryptedFileStream = new ByteArrayOutputStream();
            Cipher cipherRSA = Cipher.getInstance(FileUploadDownloadConstants.RSA_TRANSFORMATION);
            cipherRSA.init(Cipher.ENCRYPT_MODE, publicKey);
            byte[] secretKeyBytes = cipherRSA.doFinal(secretKey.getEncoded());
            encryptedFileStream.write(secretKeyBytes);
            encryptedFileStream.write(initializationVector);

            //call processFile to encrypt the file
            Cipher cipherAES = Cipher.getInstance(FileUploadDownloadConstants.AES_TRANSFORMATION);
            cipherAES.init(Cipher.ENCRYPT_MODE, secretKey, ivSpec);
            processFile(cipherAES, fileStream, encryptedFileStream);
            encryptedFileStream.close();
            logger.logExiting(METHOD_NAME);
            return encryptedFileStream;             
        }catch(Exception e) {
            e.printStackTrace();
        }
    }

This is the stack trace I'm getting :

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at java.io.OutputStream.write(OutputStream.java:75)

I'm getting the out of memory error in the line if ( tempOutputBuffer != null ) outputStream.write(tempOutputBuffer);

Is there anything wrong with the way of writing to output stream? And this happens when I'm trying to larger files like around 15 Mb. Would greatly appreciate your help and thanks in advance.

How do you invoke this function? What OutputStream do you pass — Thiyagu, Apr 23 '18 at 15:12
At first glance, what you are doing looks correct as far as the streaming implementation is concerned. But be aware that ByteArrayOutputStream writes the content inside the heap memory. Therefore, as long as you hold on to this variable, you have a large (e.g. 15MB) chunk of memory that stays in memory. This may add up. What are you doint with the returned `encryptedFileStream` that retains this memory chunk ? And where does the InputStream come from ? — GPI, Apr 23 '18 at 15:28
I see InoutFileStream is an argument for encryptFile. How many files do you open at one time? — Jake, Apr 23 '18 at 15:28
Have you tried increasing your heap like [this](https://stackoverflow.com/questions/6452765/how-to-increase-heap-size-of-jvm/6452812) example? — Jacob, Apr 23 '18 at 15:32
Your code copies file contents to the byte array hold in the memory. However, 15Mb is not that much. Do you process many files at once/in parallel? — lexicore, Apr 23 '18 at 17:35
Yes, this code processes only the stream for one file at a time. — Aravind S, Apr 24 '18 at 03:50
@GPI the encrypted stream is what I'm returning to the calling function, which will in turn store the encrypted stream of the file into the file system. — Aravind S, Apr 24 '18 at 03:54
@Jake In cases where I need to encrypt many files, I just pass the stream of one file, encrypt it and then proceed by sending the stream of next one. — Aravind S, Apr 24 '18 at 04:33
I'm guessing you still have reference to previously processed files or streams elsewhere in your code. — Jake, Apr 24 '18 at 05:54
@AravindS anyway returning an outputstream makes no real sense (unless the caller also writes to it). You should either accept an output stream to write to,as an argument, or return a plain byte[]. Increasing the heap sure will help, but knowing where the memory is hold onto is a must from a programmer’s perspective. Please trace what the input stream is and when it is garbage collected, and same for the output. — GPI, Apr 24 '18 at 07:00
@GPI actually the front end needs an output stream. I guess the references of these variables are stored in the heap through out the loop where the encryption process. — Aravind S, Apr 24 '18 at 08:19
@AravindS you may have no chance but to comply with what "the frontend" asks. But what are the methods of OutputStream ? `write()`. What can the frontend do with `write` ? Nothing, because surely, it does not write-over/after the encryption process. So 99% chances what it does is `toByteArray`. Which implies, what the front actually wants are bytes, not streams. Returning a `ByteArrayOutputStream` is kind of a code smell 1) because it's an implementation, not an interface, 2) because returning an outputStream only makes sense when you want to call `write()`. Which a frontend, I bet, does not. — GPI, Apr 24 '18 at 08:29
@GPI You're right. And wait, I hadn't completed the comment correctly, this method what I've written will only encrypt and return the encrypted stream. The calling method will use this output stream and store it in the file system. I was wrong previously, sorry about that. — Aravind S, Apr 24 '18 at 08:38

score 1 · Answer 1 · answered Apr 23 '18 at 15:28

1

Have you tried setting the minimum heap size for your JVM? See this related question:

What are the Xms and Xmx parameters when starting JVMs?

The default value might be too low for your example.

answered Apr 23 '18 at 15:28

Greg Brown

3,168
1
27
37

1

Ya, I just researched on it, but I'm trying to run this program without modifying the configurations as much as possible. – Aravind S Apr 24 '18 at 04:56
In that case, you might want to consider writing to a FileOutputStream instead because that won't consume as much heap memory. – Greg Brown Apr 24 '18 at 11:49

java.lang.OutOfMemoryError: Java heap space on writing to outputstream

1 Answers1