MSBuild assigns wrong TargetFrameworkSDKToolsDirectory

Question

A fresh dev machine fails to build our .net-framework-3.5-project on VS2019. Here's a dump of our investigation so far.

1. Diffing a verbose log from this machine against a working machine, we get this line early in the log, only on the failing machine:

Property reassignment: $(TargetFrameworkSDKToolsDirectory)="C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.7.2 Tools\" (previous value: "") at C:\Program Files (x86)\Microsoft Visual studio\2019\Professional\MSBuild\Current\Bin\ Microsoft.NETFramework.CurrentVersion.props (90,5)

2. As a result (?) many times later in the log the failing machine shows

TargetFrameworkSDKToolsDirectory = C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.7.2 Tools\

While a working machine shows

TargetFrameworkSDKToolsDirectory = C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\

3. Looking into Microsoft.NetFramework.CurrentVersion.props, the relevant lines show -

<TargetingClr2Framework Condition="'$(TargetFrameworkVersion)' == 'v2.0' or '$(TargetFrameworkVersion)' == 'v3.0' or '$(TargetFrameworkVersion)' == 'v3.5'">true</TargetingClr2Framework>
<MSBuildManagedCompilerPath Condition="'$(TargetingClr2Framework)' == 'true'">$(MSBuildFrameworkToolsRoot)\v3.5</MSBuildManagedCompilerPath >
<TargetFrameworkSDKToolsDirectory Condition="'$(TargetingClr2Framework)' == 'true'">$(SDK35ToolsPath)</TargetFrameworkSDKToolsDirectory>

<!-- If the sdk path is not 3.5 or lower set it to the 40 sdk tools path. This will allow future target framework versions to use the 4.0 sdk tool set
         When a new windows SDK revs they will inplace update the location pointed to by this property. When a new sdk is release this target will have to be
         revised along with another toolsversion. -->
<TargetFrameworkSDKToolsDirectory Condition=" '$(TargetFrameworkSDKToolsDirectory)' == '' ">$(SDK40ToolsPath)</TargetFrameworkSDKToolsDirectory>

So it seems like for some reason the failing machine didn't catch the TargetFrameworkVersion value of this project as v3.5.

4. Not sure the choice of a toolset should have made any difference, but it does (probably a different bug).

5. The MSBuild registry keys and VS2019/MSBuild folders are identical on working and failing machines.

Before we go reinstalling multiple VS versions - does anyone have an idea what might be causing it or how to diagnose deeper?

Answer 1

I'm dumping my findings here, in case it helps someone some day.

MSBuild is open source, and skimming it it seems to contain many, many extra tracing facilities. They're triggered by environment variables - probably since they're intended to be used across multiple MSBuild processes. Here's a (partial) list of ones that were useful for me:

1. MSBUILDDISABLENODEREUSE=1 - avoids spawning multiple processes, helps debugging.
2. MSBUILDENABLEDEBUGTRACING=1 - enables multiple internal Trace.WriteLine calls
3. MSBUILDDEBUGEVALUATION = 1 - some more Trace.WriteLines, around some Project property evaluation methods(IsDirty, Set\ Remove GlobalProperty)
4. MSBUILDDEBUG = 1 - enables many Console.WriteLine calls, mostly around cache management.
5. MSBUILDDEBUGONSTART =2 - waiting to attach debugger, =1 - launch new debugger.
6. MSBUILDDEBUGSCHEDULER=1 - lots of high detail messages, around BuildRequest management. Dumped to EngineTrace_{processId}.txt file, by default in Path.GetTempPath() and can be overriden by MSBUILDDEBUGPATH.

There are probably many ways to intercept these messages even without building from source. What I did was load the MSBuild executable into VS and run it - so (1) I saw all the messages in the output pane, (2) whenever I broke into a function VS downloaded the matching CS source for me (you need to enable source server + source link for this, under Tools/Options/Debugging/General).

I also did a lot of diffing of message dumps between a working machine and the failing one. In our case the problem was a bad uninstall of a previous VS build, but that probably has less interest to you.