(I know this is an old question, but my typically-Java-based team is doing a bit of .NET work and asking the same questions right now, and here's what we've found.)
If you're using a DVCS system like Git, then you're absolutely right that you may experience performance problems if you check those libraries into source control. As a point of reference, we transitioned several large projects (2-5GB) with checked-in binaries from Perforce to Git. The performance of the imported Git repos (using Git 1.9 on beefy Windows boxes using SSDs) was unsuitably slow for development. We tweaked our build to pull most of those dependencies from a private Nexus instance, and the significantly slimmer repos (50-200MB of source) seem to be performing well.
If you already have a Nexus instance available to you, there's nothing stopping you from using it to store .NET artifacts -- as far as Nexus is concerned, an artifact is just a file. If you zip up your DLLs and config files and whatnot into a single file, Nexus is happy to host that as a versioned artifact, and you can download/unzip it to the right spot when you need it. (I haven't used Artifactory, so I can't comment on what it does.)
If you want something that specifically integrates with VisualStudio (or MonoDevelop), then NuGet seems to be the emerging answer now.
By default, there's a central NuGet feed with no access restrictions on reading. For hosting, it looks like you can submit OSS/public binaries to be hosted there, and if you want to host proprietary/private binaries, there are instructions for setting up your own private NuGet feed.
If you have the Pro version of Nexus, it claims to host .NET artifacts and allow access through NuGet, but I have no experience using it.