2

I've create a trie tree with an array of children. When deleting a word, I set the children null, which I would assume deletes the node(delete is a relative term). I know that null doesn't delete the child, just sets it to null, which when using a large amount of words it causes to overflow the heap.

Running a top on linux, I can see my memory usage spike to 1gb pretty quickly, but if I force garbage collection after the delete (Runtime.gc()) the memory usage goes to 50mb and never above that. From what I'm told, java by default runs garbage collection before a heap overflow happens, but I can't see to make that happen.

trincot
  • 317,000
  • 35
  • 244
  • 286
Nicholas
  • 7,403
  • 10
  • 48
  • 76
  • 1
    you shouldn't be playing around the edges of the memory, fix your application to prevent the 1GB spike first. – mohdajami Mar 12 '10 at 00:18
  • @medopal: While valid that one should take care not to request more resourced than really needed, for some applications one really should "play around the edges of memory" in the sense of having a deep understanding of GC and leveraging that understanding to get the most from available memory. While it's "generally" bad practice to call Runtime.gc(), there is a reason it's exposed to the developer. – Eric J. Mar 12 '10 at 00:21
  • @Eric, good point, personally my Java apps could be used on desktops up to mainframes (passing by many OSes) and different setups, managing memory could become very complicated – mohdajami Mar 12 '10 at 00:36
  • DO NOT force java to swap. Very Very bad things happen, it will nuke your PC. I've actually had to reboot--it touches all the parts of it's memory way too often. – Bill K Mar 12 '10 at 00:55
  • @medopal: I suspect the vast majority of Java programs meet criteria similar to yours. Just saying there are edge cases out there in software development where you want to be able to maximize your use of available memory. – Eric J. Mar 12 '10 at 01:01
  • @Bill K: Java doesn't actually swap AFAIK, but the OS may (probably will) to meet the request for memory issued by the Java runtime. Low memory situations can sometimes force a reboot on some platforms, but I think it's an OS issue and not a Java issue directly. – Eric J. Mar 12 '10 at 01:03
  • @Eric J.: the OP is talking about Linux. Linux doesn't reboot for no reason. My hardened Linux Debian *workstation* has reached 6 months of uptime several times (critical security fixes affecting my hardened Debian aren't exactly common ;) And that is just a workstation. In addition to that, when Linux really runs out of memory it shall kill one process (or more). And if a "kill -9" isn't releasing *all* the memory used by the process, then the Un*x OS is not compliant. So, yes, this is a (broken) OS issue. Correctly designed OS don't reboot when they get low on memory. – SyntaxT3rr0r Mar 12 '10 at 02:09
  • @WizardOfOdds: Yep I agree 99.5%. My team operates around 20 Linux instances on AWS. Once in a blue moon SpringServer seems to hang up badly and kill -9 doesn't correct the problem (process is gone, but upon restarting Spring we have the same issue. A reboot clears it up. Still, that's 20 machines x around 2 years and we have had to reboot 2 times that I know of. – Eric J. Mar 12 '10 at 02:57
  • @Eric J: I meant having so much allocated to Java that your OS has to swap your java process. Once you do that, your system can become so fried swapping in and out that you can hardly regain control of it. On a linux machine you can probably SSH in or open an non-X terminal and kill it, but it may be so busy that you can't do anything with the GUI and have to power down the machine on a windows machine. – Bill K Mar 12 '10 at 17:20
  • What does "overflow the heap" mean? – Raedwald Mar 31 '16 at 16:04

5 Answers5

5

(this too long for a comment)

Contrarily to popular belief you CAN really force a GC in Java but this is not done using System.gc(). The way to really force a GC is to use JVMTI's ForceGarbageCollection() call. Don't ask me more, I asked a question here and nobody found it interesting (no upvotes) and nobody could answer it, yet JVMTI's ForceGarbageCollection() is how a lot of Java programs like IntelliJ, NetBeans, VisualVM, Eclipse etc. do really force a GC:

Java: How do you really force a GC using JVMTI's ForceGargabeCollection?

Now... You probably do not want to do that and you probably do not want to hint the GC using the "no guarantee" System.gc() call.

At how many words do you start having problems? There are very compact data structure when you need to work with insane number of words. Are you sure you're using the correct data structure and are you sure you're not having leaks?

Community
  • 1
  • 1
SyntaxT3rr0r
  • 27,745
  • 21
  • 87
  • 120
  • 2
    interestingly, one person insulted me for that question I asked, saying I was lazy... Even tough that same person has reached 20K rep answering much easier questions, asked for sure by people who did much less research than what I did on JVMTI. I flagged his comment insulting me as offensive and welcome anyone reading that question to do the same. Also if you think that knowing how to really do a GC would be a great addition to GC, than please upvote my question. – SyntaxT3rr0r Mar 12 '10 at 01:57
  • The easiest way to build up massive rep is actually by asking really easy questions in certain domains that a lot of people like to answer. I suspect I could amass tens of thousands of points of rep just by posting reworded questions from the comp.lang.c FAQ in a randomized order. – Nate C-K Feb 06 '12 at 21:54
  • Here +1. Don't bother about them. – user568109 Dec 26 '13 at 06:51
3

Are you are referring to the memory not being freed to the OS - i.e. top and similar programs show that the Java process takes 1GB of memory? Even though Java's garbage collector frees the memory from its heap, it can still keep hold of the memory so that future allocations don't need to request for more memory from the OS.

To see how much heap space is actually used by the Java objects, use VisualVM or a similar Java-specific tool. If your machine has lots of memory, then the JVM will use it (IIRC, especially the Server VM is tuned to reserve more memory), but you can always limit it with the -Xmx and other JVM options.

Esko Luontola
  • 73,184
  • 17
  • 117
  • 128
  • [Full GC 815615K->815615K(932096K), 1.6976420 secs] [Full GC 815615K->29792K(566272K), 0.2920610 secs] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3209) at java.lang.String.(String.java:216) Is the exact error without explicitly stating garbage collection. I don't really want to increase the limit, I'd like to actually address the problem. I guess the problem is how do you delete arrays that aren't needed any more? – Nicholas Mar 12 '10 at 00:28
  • If you get such an error, then maybe you have a reference leak somewhere. Try using a memory profiler to find out whether the program is keeping hold of references to unused objects, so that the GC can not free them. – Esko Luontola Mar 12 '10 at 10:58
1

OK, you are getting a java.lang.OutOfMemoryError: Java heap space.
Most probably, Runtime.gc() won't help, because if it would, the JVM had already did a gc.

It's probably a memory leak. If I were you, I will review my code carefully and see if some reference is still hold by something.

So the grandchildren aren't deleted? When I do my delete, I simply set it's child node to null, but not the children's children, but those children weren't ever initialized, only create(Node[] children = new Node[26]

If you do children=null , yes, the entire array should be gc'd. Provided that you didn't gave that reference to something.

But who knows what the culprit is. It may not even these "children" Nodes. You might want to use visualVm and find out what object is accumulating. You can use more sophisticated tools like JProfiler and examine the references etc., but if you are simply building a trie, I guess it's simpler to walk-through your code and spot the leak.

Enno Shioji
  • 26,542
  • 13
  • 70
  • 109
0

Memory allocated to a process (i.e. the JVM) is not necessarily given back to the OS in Unix. So even though the Java Virtual Machine may have fully garbage collected the heap, the process size may stay the same.

Ordinarily, this may not have much impact as the unused heap will be paged out and not paged back in again. Look at the difference between the Virtual Size (VSZ) and the Resident Set Size (RSS) in the output of ps -u the difference is how many pages are swapped out.

msw
  • 42,753
  • 9
  • 87
  • 112
0

An object will only be deleted after it can no longer be reached by links from any accessible object. Is it possible that you still have references to the objects concerned?

By the way, Runtime.gc() is sometimes only a hint that garbage collection should run.

crazyscot
  • 11,819
  • 2
  • 39
  • 40
  • So the grandchildren aren't deleted? When I do my delete, I simply set it's child node to null, but not the children's children, but those children weren't ever initialized, only create(Node[] children = new Node[26]). – Nicholas Mar 12 '10 at 00:41