There are a number of things to consider. There are two real limits for memory, first is the limit of maximum addressible memory and second is the total available virtual memory in your environment.
1) The virtual memory is (usually) made up of RAM plus swap space (disk-backed virtual memory). This memory pool is used by all processes running on your machine (including the OS). The total memory used must be less than the total available virtual memory (attempts to exceed this limit will result in errors in one or more processes).
2) The maximum addressible memory is a per-process limit on the amount of memory that can be mapped to that process. Some of the per-process memory map may be reserved for the OS/kernel. The main factor for what will be available is whether the process is 32- or 64-bit. Typically 64-bit processes don't need to worry about the limit (yet ;)) as it is very high (although this does depend on architecture).
Threoretically, 32-bit processes can address up to 4GigaBytes and 64-bit processes can address up to 16ExaBytes (in practice most 64-bit processors do not support the full amount).
Let's talk 32-bit from now on...
The OS/kernel will usually reserve a chunk of address space for its own use. For example, by default Windows reserves 2GB of address space (although there is a switch that will reduce this to 1GB). The Linux kernel will usually reserve 1GB (although this can be reduced to almost nothing with a kernel change).
So, this leaves you with 2GB on Windows 32-bit and 3GB on Linux 32-bit for your Java VM. Consider that there is a bunch of memory the JVM needs that is NOT part of the Java Heap, so your heap will have to be somewhat smaller than these amounts - how much will vary depending on your application and the JVM implementation.