That is not very clear from documentation I read, but there is a clue in the source code documentation:
/// If this property is true then the task will take every C# or VB
/// compilation which is queued by MSBuild and send it to the
/// VBCSCompiler server instance, starting a new instance if necessary.
/// If false, we will use the values from ToolPath/Exe.
It seems that reusing the same service minimizes build time since the compiler can use past build intermediates and results, and compilation results from other related projects.
Some more clues from someone on the project (Jared Parsons, Microsoft):
What's happening is the MSBuild property UseSharedCompilation
is being set to false. As such we are not using the compiler server and you're paying the JIT cost for CSC on every build. That is the reason for the slow down.