2

I've got a very weird one.

I have about a dozen PC's in our development department that all suffer the same issue: commandline builds using msbuild4.0 (VS2010) are way slower than should be. 4x to 5x slower as expected.

All machines are HP z400 Workstations (quad-core Xeon+hyperthreading, 2.4 GHz, 6GB RAM) running Windows 7 Pro 64bit, or HP Elitebook notebooks (Core-i7 4GB RAM) also on Win7 X64. If I take one of those with the vanilla factory pre-installed Win7, install VS2010 and do a build they are as fast as expected. If I install them with our company standard software image, they became 4x to 5x slower on the same VS project.

The same company software image on Lenovo laptops (Core-i5) or desktops (Core-i7) shows no measurable difference between the company image and the factory pre-installed Win7. It's even stranger: If I install VirtualBox with a Win7 image on a HP system with the problem the virtual machine DOESN'T have the problem.

Every benchmark tool I have tried shows no measurable difference between the company image and the pre-installed Win7. Only msbuild is affected and only on the HP machines running Win7 on the company image.

Before you ask: I have disabled all software/background processes in the company image, but that doesn't make any difference. It's obviously not something running in the background that somehow interacts with msbuild. My best guess is that on the HP hardware some settings gets changed that affects msbuild. This doesn't happen on other hardware. (And the VS2010 GUI is not used/executed at all. I am aware this can interact with the msbuild if both try top access the same solution/files. Antivirus doesn't make any difference either.)

Anybody have any idea what could possible influence msbuild to slow down? Any suggestion, no matter how far-fetched, is welcome.

Tonny
  • 671
  • 5
  • 17
  • Aliens. Also, you're so so so almost off topic its causing my teeth to itch. –  Mar 14 '11 at 12:22
  • Try changing the MSBuild output verbosity to diagnostics - it gives you loads of information about what is happening during a build. You *might* get some more info from that. – adrianbanks Mar 14 '11 at 12:23
  • When checking the performance effects of antivirus, it's not sufficient to turn it off. You have to uninstall it. – Ben Voigt Mar 14 '11 at 12:46
  • @Ben - Why do you have uninstall. I would assume that if its not scanning when turned off then its the same as not being there? – Ritch Melton Mar 14 '11 at 12:55
  • @Ritch: Depends on the particular program, but most of them still hook a large number of kernel functions. The hooks don't do much when the AV is set to disabled, but hooks that don't do much are very different from no hooks at all. – Ben Voigt Mar 14 '11 at 12:57
  • @Ben: I installed the same AntiVirus (TrendMicro by the way) on the machines WITHOUT the problem: Doesn't slow them down at all. – Tonny Mar 14 '11 at 13:00
  • @AdrianBanks: Tried that. Nothing significant. Seems the .NET 4 assembly handling is the part that is really slower, but WHY remains the big question. – Tonny Mar 14 '11 at 13:02
  • @Will - please scratch your itch. – Hans Passant Mar 14 '11 at 13:24
  • Ton, what are the hard drive specs for the machines in question? Hard drive performance can have a very significant impact on build times. – James Nail Mar 14 '11 at 15:47
  • @James - If I install VirtualBox with a Win7 image on a HP system with the problem the virtual machine DOESN'T have the problem. - It sounds like the HDD isn't an issue. – Ritch Melton Mar 14 '11 at 21:05
  • Ah, well sorry then, Ritch... that's all I've got on this issue. Good luck. – James Nail Mar 15 '11 at 15:29
  • @Tonny, now that you have the solution, could you please clarify the question title to include information about the buggy version of Entrust? – Peter Mularien Oct 10 '13 at 15:47
  • 1
    @PeterMularien It isn't really buggy. Just a stupid implementation that performs awfully on multi-core machines. Applies to all V9 versions and (I'm told) also earlier versions. It is fixed in V10 and later. (As far as I know the awful performance is still there, but it only gets called when needed. No longer on each certificate check. In our case only for email certificate checks and Outlook is pretty much single-threated anyway so nobody notices anymore.) – Tonny Oct 10 '13 at 18:42

4 Answers4

4

Firstly I assume you are not doing anything daft like building on a network location or mapped drive, or redirected "Documents" folder?

Next step is to use Task Manager to see which processes are slowing you down. Is processor usage at 100%? What processes are active? Is it paging a lot (look at "page faults delta")? Which processes?

I have to say this is sounds like antivirus. You should exclude OBJ, LIB, ILK, PDB and similar file types from antivirus on-access scanning.

I know you said "Antivirus doesn't make any difference" but you haven't confirmed that you have tried turning it off altogether.

Also look at search indexer, continuous backup or similar processes which might be responding to the build activity.

Ben
  • 34,935
  • 6
  • 74
  • 113
  • Hi Ben: we specifically created a local build test-scenario. Systems have ample RAM (6G, about 1.5 in use). CPU load doesn't go beyond 25% (on a quad core with hyperthreading). And we do use msbuild /M. – Tonny Mar 14 '11 at 13:04
  • As for the antivirus, see above: We tested with and without. Makes no difference: and the regular development files extensions are excluded anyway. – Tonny Mar 14 '11 at 13:06
  • Paging? IO reads/writes? To ensure it isn't doing anything at all with network resources, try unplugging it... – Ben Mar 14 '11 at 13:27
  • 1
    My builds went from 3-5 seconds for 1 solution with 15 projects to 3 minutes. Thanks for the tip here becase Microsoft Security Essentials' Real-time protection was the culprit for me. Not sure why it suddenly slowed down since MSE was installed all along but adding an exclusion to the real-time protection list solved it for me. Back down to 3-5 seconds. – Bryan Migliorisi Oct 11 '12 at 04:02
  • @BryanMigliorisi - the exact same thing applies to me. Problem is, I can only get the speed back by unchecking that "Monitor file and program activity on your computer" option. Excluding locations, file types or even processes has not worked for me so far. – Alexandru Marculescu Jan 09 '14 at 09:30
  • @AlexandruMărculescu is it possible you have not excluded the proper locations? Perhaps removing MSE and reinstalling it could help too. – Bryan Migliorisi Jan 09 '14 at 14:22
  • @BryanMigliorisi - including the sources' location doesn't work for me, I can't think of anything else; neither does reinstalling the antivirus, but I can manage – Alexandru Marculescu Jan 10 '14 at 09:18
2

Answering this myself as I have found the problem. The computers in question all have the "Entrust Entelligent Security Provider" software installed. (A certificate manager our company uses for encrypted mail handling.)

.NET4 (in which VS2010 is written) and the compilation of .NET4 code require a large amount of certificate checks and this software (which get's called in the process) apparently has issues on multi-core PC's. The more cores how slower things become.

The core-i5 systems have the problem as well, but not that badly. They are still usable. The i7 and Xeon system (8 cores) slow down to the point of becoming nearly unusable.

Unfortunately I can't get rid of this Entrust software. It's part of the company mandated software suite. I just hope the supplier has a solution for this.

EDIT: Problem is solved with version 10 of Entrust Entelligence. It does not set itself by default as first certificate handler in the system anymore.
On v9 a manual edit of the registry is required to put the MS handler back in default position.
That was the real problem: Every certificate check went first through the Entrust DLL which pushed everything in a queue which got emptied by only a single-thread/CPU. Then it discovered it wasn't supposed to deal with this certificate and handed it of to the Microsoft handler.

Tonny
  • 671
  • 5
  • 17
  • 1
    Great answer. For those who are curious, the registry key that I changed to apply this "fix" was `HKLM\SOFTWARE\Wow6432Node\Microsoft\Cryptography\Defaults\Provider Types\Type 001`. Change the value of `Name` to `Microsoft Enhanced Cryptographic Provider v1.0`. Try launching VS2010 **before exiting regedit** -- if you make a mistake here you may make it impossible to launch certain applications. – Peter Mularien Oct 10 '13 at 15:46
  • Great addition Peter. I should have supplied this myself back in the day. The tip about trying VS2010 before closing regedit is also a great idea. – Tonny Oct 10 '13 at 18:37
1

Run MSBuild from the command line with the verbosity set to diag:

MSBuild mysolution.sln /v:diag

That will produce a list containing the amount of time spent per task. That should give you some information to go on.

Ritch Melton
  • 11,498
  • 4
  • 41
  • 54
  • Tried that: It is mainly the .Net 4 assembly handling, but why this is different on these HP machines while the nearly identical Lenovo systems don't suffer it? The software installation (apart from some drivers) is completely identical on these machines. – Tonny Mar 14 '11 at 13:09
  • You mean the ResolveAssemblyReferences task? What paths is it searching? – Ritch Melton Mar 14 '11 at 13:12
  • @Rich: Just the local paths for the .NET4 SDK adn the application specific ones that are part of the build itself. And these are the same on all the PC's involved. – Tonny Mar 14 '11 at 15:37
0

A diagnostic report will show how MSBuild is working at a detailed level, but won't always help diagnose performance issues. Try running a normal build adding the following additional options to the msbuild command line:

/ds /clp:performancesummary

This will output a diagnostic summary about node usage, and also add a detailed performance summary to the end of the console output of the build, showing how long each task is running. If you use a file logger you can add the ";performancesummary" option to the end of your file logger parameters (/flp:...).

One other thing to keep an eye on would be to reduce the AssemblySearchPaths property to use only the paths you are using in your project, it has been a cause of performance issues on some builds I've seen. Just search in Microsoft.Common.targets, redefine it in a standard import used by all your projects (you do have one of those, right:) and redefine it prior to where the standard files get imported, removing any of the {...} items that you don't make use of.

Brian Kretzler
  • 9,748
  • 1
  • 31
  • 28
  • Actually, diag does all that plus show the directories being searched for ResolveAssemlbyReferences. Not that your options are bad, they just didn't show all the information I was curious to know about. – Ritch Melton Mar 14 '11 at 20:50