Typically Out of Memory Exceptions indicate something is not being properly disposed. Make sure anything you are using that is IDisposable is getting disposed when you are done with it, and you are not holding on to these resources longer than you absolutely need to be.
The best way to track down these issues is with the help of a Memory Profiling tool. The general idea is to take snapshots of memory usage and compare them, trying to find things that are holding on to memory that shouldn't be. I've used several tools, including DotTrace, ANTS profiler, .Net Memory Profiler, and they all have various strengths and weaknesses, but they generally provide a way to compare snapshots so you can see the deltas. More importantly they all have free trials, so you should be able to install one and start digging in to your problem.
If your interested, there is a StackOverflow Question about which profiler is best that while surly opinionated, should provide some good information about various tools.