Memory issues/leaks on encrypting/ decrypting large files multiple times with javax.crypto library using Sun Java Cryptography Extension (JCE)

Question

I am using javax.crypto package to encypt/decrypt files but the problem is that once a big file (around 100- 700 mb) is encrypted there is spike in memory of 70 Mb (first time) and whole of this memory is not released after execution is finished. I have kept my application run for days but this memory do not come down. Interesting thing is if I encrpyt/ decrypt the same file again and again the memory do not rise by 70 Mb, but for first 3-4 iterations 5-8 Mb of memory is released in each iteration and after that memory starts increasing again in chunk of 2-5 Mb and after few iteration some memory get released but in all the memory always increases.
The code to encrypt file is simple

 static Cipher c;// = Cipher.getInstance("AES/CBC/PKCS5Padding");
private static void Encrypt_File(String infile, String outFile) throws Exception
{
    //String destKey = "123456";
    //byte[] IV = generateRandomBytes(16);

    //byte[] salt = generateRandomBytes(16);
    //Rfc2898DeriveBytes rfc = new Rfc2898DeriveBytes("123456", salt, 1000);
    //SecretKey key = new SecretKeySpec(rfc.getBytes(32), "AES");
    SecretKey key = new SecretKeySpec(generateRandomBytes(32), "AES");
    //if(c == null)
    c = Cipher.getInstance("AES/CBC/PKCS5Padding");

    c.init(Cipher.ENCRYPT_MODE, key);
    FileOutputStream fos = new FileOutputStream(outFile);
    CipherOutputStream cos = new CipherOutputStream(fos, c);
    FileInputStream fis = new FileInputStream(infile);
    try
    {
        int len = 0;
        byte[] buf = new byte[1024*128];
        while((len = fis.read(buf)) != -1) {
            cos.write(buf, 0, len);
            //cos.flush();
        }           
    }
    finally 
    {
        c.doFinal();
        cos.flush();
        cos.close();                
        fos.flush();
        fos.close();
        fis.close();

    }
}

This is simple observation I have seen in my program: I am using Windows 7 64 bit with 16 GB RAM Intel Core 2 Duo 3.00 GHz and file encrypted was 700 MB size.

Explanation                Memory Usage(Shown in Windows Task Manager Private Working Set column)
When program starts                    9924 K
After first iteration of encryption    81,180 K
Second Iteration                       78,254 K
3 Iteration                            74,614 K
4 Iteration                            69,523 K
5 Iteration                            72,256 K
6 Iteration                            70,152 K
7 Iteration                            83,327 K
8 Iteration                            85,613 K
9 Iteration                            95,124 K
10 Iteration                           92,698 K
11 Iteration                           94,670 K

I kept the iteration on for 2000 iteration, the same pattern was observed and at the end memory usage 184,951 K, this memory was not released after calling System.gc() also.
What could be the possible problem, is it the CipherOutputStream or Cipher class having some memory leak or I am doing something wrong here?

EDIT After seeing the link (posted in comment), I made changes so that I can print what the memory usage was in JVM i.e. added these lines in code

System.out.println("  " +i +" \t\t\t " + ConvertTOMB(Runtime.getRuntime().totalMemory()) +" \t\t "+ ConvertTOMB(Runtime.getRuntime().freeMemory()) +" \t\t "+ ConvertTOMB(  Runtime.getRuntime().totalMemory() -Runtime.getRuntime().freeMemory()) );

Observed after 1000 iterations the memory usage increased and then did not returned to normal, as in first few iteration the memory usage was 1-5 MB but after 1000 iteration the memory consumption never returned to single digit, it was ranging 25-225 M.B.

Use a profiler like jProfiler, or at least take heap dumps, and use Eclipse Memory Analyzer to see what is on the heap. Without that, it's always just stabbing in the dark... — ppeterka, Oct 02 '13 at 11:50
How do you know that crypto is responsible and not some other part of your code. Might it be, eg, that you are interning a large number of Strings? — Hot Licks, Oct 02 '13 at 12:05
@HotLicks this same encrypt function is called in while loop for testing and that too on the same file. — Deepak Bhatia, Oct 02 '13 at 12:12
I doubt this is related to your memory problem but calling `cos.flush()` inside the loop is almost certainly wrong. — President James K. Polk, Oct 02 '13 at 14:41
@GregS I have removed cos.flush() inside the while loop too but no difference — Deepak Bhatia, Oct 02 '13 at 16:00
possible duplicate of [Java still uses system memory after deallocation of objects and garbage collection](http://stackoverflow.com/questions/324499/java-still-uses-system-memory-after-deallocation-of-objects-and-garbage-collecti) — Henry, Oct 04 '13 at 09:35
@Henry in the above mentioned question the memory is not always increasing and I have made one more test project where I am reading the same 700 mb files in while loop and I am writing again to disk without encrypting the file the memory usage remains the same — Deepak Bhatia, Oct 04 '13 at 10:38
@Henry in that question it is said that the JVM will reuse the memory which has been garbage collected but in my case the JVM is always allocating new memory from OS and the physical memory grows — Deepak Bhatia, Oct 04 '13 at 11:30
I have the same problem but I have much more loop than you :( My memory increase for 4-5 GB @@ — Slim_user71169, Apr 29 '16 at 10:10

score 0 · Answer 1 · answered Oct 04 '13 at 07:36

0

The JVM usually does not give back memory to the OS once it has it allocated. Instead it keeps it as free heap memory and can use it for other objects.

This means, on system level you will just see an increasing memory usage until the JVM is stopped.

answered Oct 04 '13 at 07:36

Henry

42,982
7
68
84

can you post the links where the above statement can have proof? And if this is so when the memory will get released will it always increase and you said it resuses memory but in my case always new memory is allocated. – Deepak Bhatia Oct 04 '13 at 09:24
See for example the answer to this question: http://stackoverflow.com/questions/324499/java-still-uses-system-memory-after-deallocation-of-objects-and-garbage-collecti – Henry Oct 04 '13 at 09:34
I copied the code as mentioned in the link and used while loop to assign the memory to array list again and then clear the list. After 500 such iterations also the physical memory on the task manager remained near 347 K – – Deepak Bhatia Oct 04 '13 at 11:24

Memory issues/leaks on encrypting/ decrypting large files multiple times with javax.crypto library using Sun Java Cryptography Extension (JCE)

1 Answers1