36

This morning it was reported that our web app on our QA server was completely broken with the following error reported from Web.config:

Could not load file or assembly 'System.Web.Mvc, Version=5.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified

Remembering seeing a Windows Update that mentioned MVC, I did some digging and found lots of people reporting a recent Windows Update breaking MVC.

After much digging through those questions and our server, it seems that what's bitten us does not match what's in those other questions, but it does appear related. Here's what we think know:

  • Our app that is broken uses ASP.NET MVC 5.1
  • MVC was installed via NuGet
  • Our BuildServer and QA servers do NOT have MVC 5.1 installed (therefore, not GAC'd)

What we believe has broken caused the "bad build" to be created:

  • A patch for MVC 5.1 was installed on the BuildServer via Windows Update despite not having MVC 5.1 installed in the GAC
  • The patch has put the "updated" version of MVC 5.1 in the GAC
  • CopyLocal=true is ignored when a DLL is in the GAC; therefore since the patch, this means that builds of our app from the BuildServer no longer have System.Web.MVC in the output folder
  • Since System.Web.MVC is not in the GAC on our QA servers (they have not yet been patched), the application now fails, because System.Web.MVC cannot be found

Assuming the behavior described above is correct, this means that any time MS service a NuGet DLL via Windows Update that we do not have in the GAC, our BuildServer will start producing incomplete builds (missing out those DLLs that have been injected into the GAC).

Upgrading to MVC 5.2 solves this issue (likely because it wasn't patched, and was therefore not injected into the GAC); the DLL is now copied to the output folder. There are no changes in the diff that upgraded to 5.2.2 except for version number changes (there's specifically no <Private> node been added/edited).

We do not wish to start GACing everything, nor creating manual build steps to copy all of our DLLs into the bin folder just in case MS patches them.

So, what can we change today to ensure we don't ever end up with out BuildServer silently producing back bad builds if MS patch other DLLs in the future?

Community
  • 1
  • 1
Danny Tuppeny
  • 40,147
  • 24
  • 151
  • 275
  • 2
    It is surprising how messy .NET versioning is. – usr Oct 20 '14 at 14:45
  • 3
    It seems that the copy local setting is missing from the project file if the default is true, setting to false and then true again would add the setting. See: http://blogs.msdn.com/b/jjameson/archive/2009/11/18/the-copy-local-bug-in-visual-studio.aspx – Guffa Oct 20 '14 at 14:54
  • 1
    @Guffa This doesn't explain why our update to MVC 5.2.2 fixed the issue. It did not introduce any changes to the project except for version number changes? – Danny Tuppeny Oct 20 '14 at 15:04
  • @DannyTuppeny: That's why I posted a "it seems" comment, not an answer. Does the property exist in your project files? – Guffa Oct 20 '14 at 15:18
  • @Guffa Not in the MVC v5 project (which misses the DLL for 5.1, but includes it for 5.2, despite no change exception version numbers in the project file :/) – Danny Tuppeny Oct 20 '14 at 15:26
  • FWIW, my pre-existing (prior to update) MVC 5.2.2 projects seem to have dodged this issue. But your overall question stands to be answered (hopefully officially by MS). I recall some effort to decouple ASP.Net releases from framework releases (Nuget) in an effort to "decouple" from it (and tooling release/VS). Seems Windows Updates need to be part of the equation.... – EdSF Oct 20 '14 at 15:36
  • It would be really helpful if you could enable Binding Fusion Log and see the details of how the loading is working. – vcsjones Oct 20 '14 at 16:42
  • @vcsjones The error is caused by System.Web.Mvc not existing (neither in the GAC nor the bin folder), so I don't think that'll reveal anything interesting. – Danny Tuppeny Oct 21 '14 at 08:25

3 Answers3

14

A patch for MVC 5.1 was installed on the BuildServer via Windows Update despite not having MVC 5.1 installed in the GAC

Yes, this behavior is actually by design. See http://blogs.msdn.com/b/dotnet/archive/2014/01/22/net-4-5-1-supports-microsoft-security-updates-for-net-nuget-libraries.aspx.

The patch has put the "updated" version of MVC 5.1 in the GAC

Yes, that's correct; it's how the patch gets the updated code to run instead of the old code. See https://technet.microsoft.com/en-us/library/security/ms14-059.

CopyLocal=true is ignored when a DLL is in the GAC; therefore since the patch, this means that builds of our app from the BuildServer no longer have System.Web.MVC in the output folder

Not exactly. What actually happens is a project that previously was CopyLocal=true gets switched to CopyLocal=false. CopyLocal can be set in one of two ways: 1) If there's an explicit <Private>True</Private> setting in the .csproj file, or 2) By default, if no such setting exists (GAC'd assemblies do not CopyLocal by default; other assemblies do).

So what appears to have happened in this case is that your project file didn't have this setting in the csproj file. As a result, the GUI showed the setting based on the evaluated default value before the patch (CopyLocal = true) but then after the patch was installed, the GUI will now show the new default value for a GAC'd assembly (CopyLocal = false).

Since System.Web.MVC is not in the GAC on our QA servers (they have not yet been patched), the application now fails, because System.Web.MVC cannot be found

That's correct.

Assuming the behavior described above is correct, this means that any time MS service a NuGet DLL via Windows Update that we do not have in the GAC, our BuildServer will start producing incomplete builds (missing out those DLLs that have been injected into the GAC).

For any .csproj reference without an explicit <Private>True</Private> setting, that is correct. Also, note the using NuGet to update your MVC reference can remove this setting even if it was previously present. See http://nuget.codeplex.com/workitem/4344.

Upgrading to MVC 5.2 solves this issue (likely because it wasn't patched, and was therefore not injected into the GAC); the DLL is now copied to the output folder. There are no changes in the diff that upgraded to 5.2.2 except for version number changes (there's specifically no node been added/edited).

That's correct. Since MVC 5.2 is not GAC'd, even without an explicit <Private>True</Private> setting, the default value of this non-GAC'd assembly will be CopyLocal=true.

We do not wish to start GACing everything, nor creating manual build steps to copy all of our DLLs into the bin folder just in case MS patches them. So, what can we change today to ensure we don't ever end up with out BuildServer silently producing back bad builds if MS patch other DLLs in the future?

The best you can do today is:

  1. Put explicit <Private>True</Private> settings in your .csproj file for all your NuGet package assembly references.
  2. Until NuGet bug #4344 is fixed, any time you use NuGet to update a package reference, go back into your .csproj file and re-add the explicit <Private>True</Private> setting.
dmatson
  • 6,035
  • 1
  • 23
  • 23
5

I believe this issue is addressed in the .Net Web development tools and UI blog here: link

I won't repeat the whole thing here, as the issue and resolution is explained pretty well at that link.

However just to repeat the key points, which should explain why this has happened:

  • As part of patch KB2994397 MVC 5.1 was added to the GAC.

  • There appears to be a NuGet bug that resets CopyLocal flag. (see link ) This means that when a machine with the above patch deploys to a non-patched machine it will break!

  • MVC 4 has had its assembly version number incremented by the same security update - MS14-059 (so the GAC version will NOT be used) This explains why the MVC 4 version still works - despite it being in the GAC.

James S
  • 3,558
  • 16
  • 25
  • I don't completely understand the last part; even if the version number of the GAC'd v4 DLL was different; if it's higher, wouldn't it still be used from the GAC? – Danny Tuppeny Oct 20 '14 at 16:56
  • 1
    (Although this answers a lot; it doesn't answer the main question, of how do we protect ourselves against this happening in the future? urrently it seems any patch from MS would cause our build server to silently create broken builds). – Danny Tuppeny Oct 20 '14 at 16:57
  • To address your first comment: Presumably the update was applied to the build server only. Therefore in your projects they would still be referencing the old MVC 4 assembly version (no longer the version in the Build server GAC), so that old version is still getting copied. When MVC4 next gets updated by Nuget in your projects I would expect the MVC4 ones to break if copyLocal is not set. Not sure on the best protection against breakage at this point - Presumably its ensuring CopyLocal is never unset! – James S Oct 21 '14 at 12:08
-2

I added a note about this issue in my blog: Microsoft Asp.Net MVC Security Update MS14-059 broke my build!.

Your analysis of the problem is right on the money, by default the Copy Local flag is set to false when the assembly is in the GAC, manually setting it to true should fix this problem.

Upgrading to 5.2.2 is even better, you get the benefits of the new release in addition to the security fix.

Miguel Lacouture
  • 249
  • 1
  • 5
  • This doesn't answer the questions about why 4 was not affected, 5.1 was, and 5.2 appear not to; nor how to protect against this happening in future if the CopyLocal flag is being reset by NuGet on every upgrade. – Danny Tuppeny Oct 20 '14 at 16:53
  • @Miguel, could you please provide a download link for the "MVC 3.0.1 tooling refresh for Visual Studio 2010"? I've googled a lot yesterday with no luck – ken2k Oct 21 '14 at 08:23
  • MVC 3.1 Tooling update: http://www.microsoft.com/en-us/download/details.aspx?id=1491 Note that there's a known issue with NuGet 1.2 and 1.3, if you have one of these versions installed in the machine you need to remove it before installing the MVC 3.1 tooling update, then you can install the latest version of NuGet. – Miguel Lacouture Oct 30 '14 at 01:41