Obviously you are using too much memory, so now you need to find out why. Without your source it is very hard to to say what is wrong, but even with source it can be problematic when the program start to become complex.
What I have found helpful is to take memory dumps and look at them in tools such as Memory Analyzer (MAT). It can even compare several dumps to see what kind of objects are allocated. When you get an idea of what objects exists which you don't think should be there you can use the tool to see what roots it has (which objects has a reference to it).
To get a memory dump form a running java program use jmap -dump:format=b,file=heap.bin and to automatically get a memory dump when your program gets and OutOfMemoryError you can run it with java -XX:+HeapDumpOnOutOfMemoryError failing.java.Program