What you will need is not only a 64bit OS and a 64bit VM, but also more memory.
On a 32bits Windows system the virtual address space is split with 2 GB for kernel operations and 2 GB for user applications. So you're screwed.
There's one possible but very unlikely workaround: you can enable the /3GB switch to raise this limitation and have the system allocate 1GB of virtual address space for for kernel operations and 3GB for user applications (if they are /LARGEADDRESSPACEAWARE).
Unfortunately, the 32bits Sun/Oracle HotSpot JVM isn't LARGEADDRESSAWARE (that I know of), and other 32bits JVM likely aren't either.
But think about it: even if you were able to do that, you would use all the memory available for you system. Nothing would be left for other programs after you've allocated your 3GB of heap for your JVM. Your system would be swapping to disk all the time. It would be unusable.
Just get a 64bis OS with more RAM. That's all there is for you, short of finding ways to have your program use less memory.