As I don't enjoy my questions to be left open when they're answered, I'll writ eup what colinsmith wrote in the comments. If he chooses to post it as an answer himself, I'll accept that answer and delete this.
Make multiple "Configurations"....on your Solution choose Configuration Manager, then "copy" your "Debug" configuration, and name it "Debug NET 4", etc, etc...use the "Batch Build" so your "Solution Build" builds each Configuration.. otherwise get a book on MSBUILD and get the MSBUILD script to output multiple targets. http://msdn.microsoft.com/en-us/library/jj651644.aspx
Conditional Compilation and Framework Targets
If you want to run/debug the NET 4 version, then just change the configuration to the "Debug NET4" one....otherwise change it to "Debug NET2" (or whatever name you decided to use to distinguish them)
This addin might lessen the pain of switching the startup project: visualstudiogallery.msdn.microsoft.com/… (use the Smart, or Most Recently Used configuration option of that addin)....then you could record two macros where you "select" the name of the "startup project", followed by the "Build" action....then just alternate between running each macro (which will change the startup project to the different ones).