This is normal behavior for Java (as well as Microsoft .NET), mostly because of their pointer model, and also their garbage collection model.
An object variable is actually a pointer to an object on the heap. In 64-bit versions, this pointer requires twice as much space. So, the pointers stored in containers will require more memory, and so will the pointers that are held by the garbage collector to allow collection. Since objects are mostly made up of pointers to other objects, the difference between 32-bit and 64-bit adds up very fast.
Added to that, the garbage collector has to keep track of all of these objects efficiently, and in 64-bit versions, the collector tends to use a larger minimum allocation size so it doesn't have to keep track of as many slices of memory. This makes sense because objects are bigger anyway. I believe the minimum sizes are typically 16 bytes in 32-bit mode and 32 bytes in 64-bit mode, but those are entirely up to the specific virtual machine you are using, so they will vary.
For example, if you have an object that only requires 12 bytes of heap memory, and you are running on a virtual machine with a 32-byte minimum allocation size, it will use 32 bytes, with 20 of those bytes wasted. If you allocate the same object on a machine with a 16-byte minimum size, it will use 16 bytes, with 4 wasted. The alternative to this is to waste a lot more memory in keeping track of those blocks, so this is actually the best approach, and will keep your application's performance and resource utilization balanced.
Another thing to keep in mind is that the Java runtime allocates blocks of memory from the operating system for its heap, then the program can allocate memory out of those blocks. The runtime tries to stay ahead of your program's memory needs, so it will allocate more than is needed and let your program grow into it. With a higher minimum allocation size, the 64-bit runtime will allocate bigger blocks for its heap, so you will have more unused memory than with a 32-bit runtime.
If memory consumption is a serious constraint for your particular application, you might consider native-compiled C++ (using actual C++ standards, not legacy C with pointers to objects!). Native C++ typically requires 1/5 of the memory of Java to accomplish the same thing, which is the reason that native code tends to be more popular on mobile devices (C++ and Objective C). Of course, C++ has its own issues, so unless you have a desperate need to reduce memory consumption, it is probably best to accept this as normal behavior and keep using 64-bit Java.