The amount of RAM in a machine has nothing to do with an OutOfMemoryException. You'll get that exception when the process runs out of virtual memory, failing to find a hole in the address space that's big enough to fit an allocation request. That usually happens when the VM size of the process starts getting close to 1.5 GB on a 32-bit machine.
Limiting the amount of virtual memory space is easy enough, just create a bunch of byte[] arrays at the start of your program and store them in a static variable. This doesn't help diagnose a memory leak at all, it merely trips the exception quicker. Use a memory profiler to find the real problem.
And do consider the possibility that this isn't a leak at all, it's not that easy to leak with a garbage collector. But just a side effect of your program processing and storing a lot of data. Which is trivially solved with a 64-bit operating system, it provides oodles of virtual memory space, limited only by the maximum size of the paging file. Not RAM.