All previous answers are correct, but I've made a test on a VM without .NET SDK/Visual Studio Installed. It is a plain 2003 server with .NET 4 Client Profile + .NET 4 Extended Framework.
And result is that in this setup CSharpCodeProvider works just fine!
If you want to protect yourself even more, you can check for specific code provider availability with this call:
CodeDomProvider.IsDefinedLanguage("CSharp")
And documentation for this method says:
The Element in the machine configuration file
(Machine.config) contains the language provider and compiler
configuration settings for each CodeDomProvider implementation on the
computer. The IsDefinedLanguage method searches the provider
configuration elements for the specified language name.
Therefore it is possible to run your program on computer where CodeProvider WILL NOT work (may be some paranoid system administrator wanted to disable dynamic code compilation (although it is meaningless because using CodeDomProvider requires full trust anyway)).