18

The following code is causing a OutOfMemmoryError: heap space for some 3 million rows.

Memory allocated to JVM is 4 GB, using 64 bit installation.

while (rs.next())
{    
    ArrayList<String> arrayList = new ArrayList<String>();
    for (int i = 1; i <= columnCount; i++)
    {
        arrayList.add(rs.getString(i));
    }

    objOS.writeObject(arrayList);
}

The memory referenced by the ArrayList is eligible for garbage collection in each iteration of the while loop, and internally JVM calls garbage collection (System.gc()) before throwing an OutOfMemoryError because of heap space.

So why is the exception occurring?

Lii
  • 11,553
  • 8
  • 64
  • 88
Nik
  • 695
  • 1
  • 4
  • 15

2 Answers2

38

Is objOS an ObjectOutputStream?

If so, then that's your problem: An ObjectOutputStream keeps a strong reference to every object that was ever written to it in order to avoid writing the same object twice (it will simply write a reference saying "that object that I wrote before with id x").

This means that you're effectively leaking all ArrayList istances.

You can reset that "cache" by calling reset() on your ObjectOutputStream. Since you don't seem to be making use of that cache between writeObject calls anyway, you could call reset() directly after the writeObject() call.

Joachim Sauer
  • 302,674
  • 57
  • 556
  • 614
  • +1 I didn't realise it did that - it seems counter-intuitive as Streams tend to be used to prevent using too much memory. –  Sep 21 '11 at 06:10
  • @Bringer128: yes, it's a non-obvious feature, but very much necessary: if it *didn't* do that, then serializing a big object tree could *very* easily end in an endless loop (think of circular references). – Joachim Sauer Sep 21 '11 at 06:11
  • One wonders whether it could be done with WeakReferences. In theory the Object that's being written cannot fall out of scope until `writeObject` returns. –  Sep 21 '11 at 06:25
  • 1
    @Bringer128: yes, that would be feasible. The reason this was not done is probably that the `ObjectOutputStream` was introduced in Java 1.1, but `WeakReference` was introduced in Java 1.2. And changing the `ObjectOutputStream` implementation to using a `WeakReference` would *surely* be a breaking change. – Joachim Sauer Sep 21 '11 at 06:31
  • Nice find! I've now deleted my answer as that link explains how writeUnshared is not deep, so is probably not useful for the OP's question. –  Sep 21 '11 at 06:42
  • +1: Its worth noting that if your receiver has less free memory, an ObjectInputStream can trigger an OOME for the same reason. (It also has to keep a reference to every object) – Peter Lawrey Sep 21 '11 at 07:42
  • @PeterLawrey: yes, but note that a `reset()` call gets recorded in the stream and will trigger the equivalent reaction in the `ObjectInputStream` as well. – Joachim Sauer Sep 21 '11 at 07:43
  • True, the solution solves what could appear to be two different problems. – Peter Lawrey Sep 21 '11 at 07:52
  • I have a question, not sure the best way to ask. Will a flush() call do the same as a reset()? Sorry, in other words, will flush, also break the strong reference? – D-Klotz Apr 16 '13 at 20:31
3

I agree with @Joachim.

The below suggestion was a myth

In addition, it is recommended (in good coding convention) that do not declare any object inside the loop. Instead, declare it just before the loop start and use the same reference for initialization purpose. This will ask your code to use the same reference for each iterations and cause less burden on memory release thread (i.e. Garbage collection).

The Truth
I have edited this because I feel that there may be many people who (like me before today) still believe that declaring an object inside loop could harm the memory management; which is wrong.
To demonstrate this, I have used the same code posted on stackOverflow for this.
Following is my code snippet

package navsoft.advskill.test;

import java.util.ArrayList;

public class MemoryTest {

    /**
     * @param args
     */
    public static void main(String[] args) {
        /* Total number of processors or cores available to the JVM */
        System.out.println("Available processors (cores): "
                + Runtime.getRuntime().availableProcessors());
        /*
         * Total amount of free memory available to the JVM
         */
        long freeMemory = Runtime.getRuntime().freeMemory();
        System.out.println("Free memory (bytes): "
                + freeMemory);
        /*
         * This will return Long.MAX_VALUE if there is no preset limit
         */
        long maxMemory = Runtime.getRuntime().maxMemory();
        /*
         * Maximum amount of memory the JVM will attempt to use
         */
        System.out.println("Maximum memory (bytes): "
                + (maxMemory == Long.MAX_VALUE ? "no limit" : maxMemory));
        /*
         * Total memory currently in use by the JVM
         */
        System.out.println("Total memory (bytes): "
                + Runtime.getRuntime().totalMemory());
        final int LIMIT_COUNTER = 1000000;
        
        //System.out.println("Testing Only for print...");
        System.out.println("Testing for Collection inside Loop...");
        //System.out.println("Testing for Collection outside Loop...");
        //ArrayList<String> arr;
        for (int i = 0; i < LIMIT_COUNTER; ++i) {
            //arr = new ArrayList<String>();
            ArrayList<String> arr = new ArrayList<String>();
            System.out.println("" + i + ". Occupied(OldFree - currentFree): "+ (freeMemory - Runtime.getRuntime().freeMemory()));
        }
        System.out.println("Occupied At the End: "+ (freeMemory - Runtime.getRuntime().freeMemory()));
        System.out.println("End of Test");
    }

}

The result from the output is clearly shows that there is no difference in occupying/freeing the memory if you either declare the object inside or outside the loop. So it is recommended to have the declaration to as small scope as it can.
I pay my thanks to all the experts on StackOverflow (specially @Miserable Variable) for guiding me on this.

Hope this would clear your doubts too.

Community
  • 1
  • 1
Naved
  • 4,020
  • 4
  • 31
  • 45
  • 2
    I'm not convinced that this will have any effect at all. In fact I *highly* suspect the compiler to re-use the same local variable slot for `arrayList` in each loop iteration, which would make it equivalent to "declare it outside the loop". – Joachim Sauer Sep 21 '11 at 06:45
  • 1
    -1 What? Recommendation is to *always* declare variable with smallest scope. Can you site any references? – Miserable Variable Sep 21 '11 at 07:12
  • @HemalPandya, While creating a coding convention for my current organization, I referred this http://weblogs.java.net/blog/ddevore/archive/2006/08/declare_variabl.html – Naved Sep 21 '11 at 07:41
  • 1
    I think this is premature optimization and quite dangerous at that. Limiting scope is has known benefits, time was when in `C` all variables were (had to be) declared at the top of the function. We don't do that anymore. See http://stackoverflow.com/questions/377763/declare-an-object-inside-or-outside-a-loop. I would urge you to change the coding convention. – Miserable Variable Sep 21 '11 at 07:54
  • I agree according to the readability, but still not sure about the memory leaks. Does initializing an object inside a loop and inside a frequently called method, not the same? – Naved Sep 21 '11 at 08:52
  • 1
    It is not related to memory leak.. In this case, `arrayList` is declared outside the `while (rs.next())` loop then `arrayList = new ArrayList();` would still create new object and the old one will become eligible to gc. – Miserable Variable Sep 21 '11 at 15:03
  • 2
    Thank you @HemalPandya. I have edited my answer; kindly have a look on it. – Naved Sep 22 '11 at 07:49
  • 5
    Good work Naved. I changed my downvote to upvote :) If there is one thing I have learned after years of programming is that we all keep making mistakes. What makes a good programmer is one who is willing to correct their mistake. – Miserable Variable Sep 22 '11 at 08:38
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/3688/discussion-between-hemal-pandya-and-naved) – Miserable Variable Sep 22 '11 at 08:38