The only differences between 32-bit and 64-bit builds of any program are the sizes of machine words, the amount of addressable memory, and the Operating System ABI in use. With Java, the language specification means that the differences in machine word size and OS ABI should not matter at all unless you're using native code as well. (Native code must be built to be the same as the word-size of the JVM that will load it; you can't mix 32-bit and 64-bit builds in the same process without very exotic coding indeed, and you shouldn't be doing that with Java about.)
The 64-bitter uses 64-bit pointers. If you have 4GB+ RAM, and are running Java programs that keep 4GB+ of data structures in memory, the 64-bitter will accommodate that. The big fat pointers can point to any byte in a 4GB+ memory space.
But if your programs use less memory, and you run the 64-bit JVM, pointers in will still occupy 64 bits (8 bytes) each. This will cause data structures to be bigger, which will eat up memory unnecessarily.