Your heap memory is only a part of the memory used by the JVM. Additionally you have native memory and permgen.
You can limit the permgen memory via command line parameters. See What does PermGen actually stand for?. In my experience, the permgen limit was defaulted to something like 1G, which was way more than we ever needed. I think we overrode it to 128m.
Native memory is a lot trickier. This is memory allocated by native libraries used directly or transitively by your code.
In jrockit, you can get a print out of a memory summary via jrcmd print_memusage. Not sure how to do that in other JVMs.
Also: http://www.ibm.com/developerworks/linux/library/j-nativememory-linux/index.html