You can use C# to build games. The question is what exactly are you intending to do? What platforms do you intend to target, and how polished do you intend the finished product to be?
Others have mentioned Unity, which uses C# and provides a ready-made game engine and development suite. The only downside is that the free version has limitations.
If you want to build your own engine for the sake of understanding, look into XNA. Or you can use a wrapper around OpenGL like SharpGL. Or maybe you can find the long-dead Managed DirectX floating around somewhere. Or if you are really brave, you can use unsafe code and wrap GDI calls so that you don't have to deal with the horribly slow GDI+ implementation. The last two really aren't recommended, and only XNA is going to provide you more than a way to draw things on the screen. There are sure to be countless other possibilities, especially considering what becomes available to C# developers with Mono.
Whatever you decide, the garbage collector isn't going to get in your way, and unsafe code wouldn't be a solution if it did.
Edit:
As mentioned by cdoubleplusgood, XNA is no longer in active development. Look into Monogame and consider the wonders of cross-platform development a bonus.