9

I have a Java console app that's processing big xml files using DOM. Basically it creates xml files from data it takes from the DB. Now, as you guess it's using large amount of memory but, to my surprise, it's not related to bad code but to "java heap space not shrinking". I tried running my app from Eclipse using these JVM params:

-Xmx700m -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20

i even added the

-XX:-UseSerialGC

as i found out that parallel GC ignores "MinHeap" and "MaxHeap" options. Even with all those options graph of my app's memory use looks like this: java app graph

As you can see, at one point my app takes ~400 MB of heap space, heap grows to ~650 MB, but few seconds later (when xml generation is done) my app goes down to 12MB of used heap, but "heap size" remains at ~650 MB. It takes 650 MB of my ram! It's bizzare, don't you think?

**Is there a way to force JVS to shrink availabe heap size to, like 150% of current used heap?**Like, if my app needs 15 MB of ram, heap size is ~20MB, when my app asks for 400 MB of ram, heap grows to ~600 MB and DROPS back to ~20 MB as soon as my app finish heavy-lifting operation?

guest86
  • 2,894
  • 8
  • 49
  • 72
  • It _maps_ 650 MB into memory, yes; it doesn't mean that it uses all of these physically. In C, I can create a 2 TB anonymous mapping using `mmap()` (on a 64bit platform), it doesn't mean these 2 TB are actually used. – fge Mar 18 '14 at 13:34
  • 1
    @PeterBratton The accepted answer to your possible duplicate is wrong. – Marko Topolnik Mar 18 '14 at 13:37
  • 3
    @Marko Doesn't matter. It's the exact same question. – Peter Bratton Mar 18 '14 at 13:43
  • 1
    @PeterBratton It matters quite a bit. You are sending a message both to OP and everyone else that the answer to that other question is correct for this question. – Marko Topolnik Mar 18 '14 at 13:46
  • 2
    @Marko If you have a better answer, please post it and I'll be happy to vote it up for you. But I don't think it's in the interest of the community to have an identical question answered differently in two places on this site. If the dupe is wrong, you should explain why there. Either way, this is a duplicate, according to the principles of stack exchange as I understand them. – Peter Bratton Mar 18 '14 at 14:09
  • @PeterBratton It is not in the interest of community to have accepted wrong answers, but that damage has already been done. You and I apparently differ in how exactly to go about mitigating it: you think your job is done when you have voted to close as a duplicate. – Marko Topolnik Mar 18 '14 at 14:12
  • 1
    @MarkoTopolnik Why don't you and Peter take it to meta. Assuming there isn't already a question regarding duplicate question with wrong accepted answer (I can't find such a question presently), this would make for a good discussion, and we can all get some solid clarification. Frankly, I think you both have valid points. – Paul Richter Mar 18 '14 at 14:33
  • Why does this worry you? It does not mean that the application is hoarding resources away from other processes; what is important is the working set after all. – fortran Mar 18 '14 at 14:39
  • @PaulRichter I actually like [this meta answer](http://meta.stackexchange.com/a/147651/186336) to a very related question. – Marko Topolnik Mar 18 '14 at 14:40
  • @fortran The larger the heap, the larger the support structures, and the larger the overhead of housekeeping. – Marko Topolnik Mar 18 '14 at 14:41
  • @PaulRichter I saw it in the meta: http://meta.stackexchange.com/questions/78438/blatantly-wrong-accepted-answer – aalku Mar 18 '14 at 14:41
  • @MarkoTopolnik I know, but with multi-level paging that overhead is minimal. Reducing the heap is also expensive, so why bother (when you might need to increase it again later on)? – fortran Mar 18 '14 at 22:15

1 Answers1

2

You should probably use Parallel collection and use -XX:-UseAdaptiveSizePolicy. Docs are for Java 1.5 but I can't find anything more recent.

The implementation of -XX:+UseAdaptiveSizePolicy used by default with the -XX:+UseParallelGC garbage collector has changed to consider three goals:

  • a desired maximum GC pause goal
  • a desired application throughput goal
  • minimum footprint

The implementation checks (in this order):

  1. If the GC pause time is greater than the pause time goal then reduce the generations sizes to better attain the goal.
  2. If the pause time goal is being met then consider the application's throughput goal. If the application's throughput goal is not being met, then increase the sizes of the generations to better attain the goal.
  3. If both the pause time goal and the throughput goal are being met, then the size of the generations are decreased to reduce footprint.

EDIT

Added "-" per OP's suggestion.

mttdbrd
  • 1,791
  • 12
  • 17
  • 1
    Yes! This did the trick! Using "-XX:-UseAdaptiveSizePolicy" and "-XX:+UseParallelGC" seems to solve the problem :) Thanks a lot! – guest86 Mar 18 '14 at 14:42
  • `UseParellelGC` should be the default (ever since Java 1.4, I believe). – Marko Topolnik Mar 18 '14 at 14:45
  • Sorry, i used "-XX:-UseParallelGC" (note the "-"). With "+" heap space is growing slowly (together with the app) and it slows down the app, with "-" it grows just the way it usually does but it shrinks in the end ;) – guest86 Mar 18 '14 at 14:55
  • I would be careful with those settings because they may actually hurt your performance. Are you sure you are not doing this just out of some abstract sense of what is "right"? – Marko Topolnik Mar 18 '14 at 14:56
  • I have a server with 4 GB of ram and several of these xml processing machines. Each one of them is separate jar file and works once per 30 sec. That's why i don't want them to be greedy :) – guest86 Mar 18 '14 at 14:58
  • @MarkoTopolnik Oracle recommends disabling UseAdaptiveSizePolicy to tune JVM performance for precisely this problem. See here: http://www.oracle.com/in/javaonedevelop/garbage-collection-mythbusters-chuk-400744-en-in.pdf (page 41). – mttdbrd Mar 18 '14 at 14:58
  • They don't exacly *recommend* turning it off; they just advise that *if tuning manually*, you should turn it off. BTW that looks like a very old presentation: they still talk about -d32 and Serial Collector being the default. – Marko Topolnik Mar 18 '14 at 15:11
  • @guest86 Your comment sounds exactly like I worry: you just feel it's *wrong* that the JVM allocates all that RAM; you don't have the hard performance data to back it up. – Marko Topolnik Mar 18 '14 at 15:13
  • Well this is the problem: Machine 1 takes 700MB of ram to create a file. Other machines sleep in the meantime. After machine 1, machine 2 wakes up and takes another 700 MB of ram to create it's file. Same for machine 3. In the end, server (not mine, i'm not an admin) will start using the swap and all machines will slow down but in reality each one of them needs 700 mb of ram only during 10 secs of work! After the task (while they sleep and await new changes) all they need is about 20 MB of ram each. – guest86 Mar 18 '14 at 15:14
  • By "machine" you mean a virtual machine, or a JVM? But anyway, that would look like you do have a real issue with heap sizing. – Marko Topolnik Mar 18 '14 at 15:19
  • Each "machine" is another "java -jar machine". – guest86 Mar 18 '14 at 15:20