24

We are developing an application framework + "plugins" using TeamCity as a CI server.

Project Details

  1. 4 Visual Studio solutions
  2. ~70 projects (and increasing)
  3. Currently running 2 builds using TeamCity: CI and FULL build.

CI - triggered on every commit.

FULL - runs nightly.

I would like to improve the performance of both builds (especially the CI build, as it needs to give its output as quickly as possible).

Are there any guidelines in general on what can be effectively and easily improved?

The build process simply builds a .sln file and runs some unit tests.

Directions Considered:

  • MSBuild parallelization
  • Overriding CopyFilesToLocal

Not sure these are applicable/will result in a performance gain.

I am looking for more ways to improve the build time (which takes around 3-4 minutes).

lysergic-acid
  • 19,570
  • 21
  • 109
  • 218

4 Answers4

14

Minimise the work your ci builds do.

  • set up the workspace with any unneeded folders cloaked, to minimise the number of files you need to get from source control. If necessary reorganise your source structure so that it's easy to knock whole folders of data out of the build with cloaking.

  • make sure the ci build uses incremental get and incremental build.

  • only build solutions/projects that you need to. If your libraries only get changed infrequently, can you precompile them and check the binaries into source control? If a project is not currently being actively developed, don't bother building it in ci builds.

  • do you need to run unit tests for every check in? We run a simple ci build of code only, with a separate test build running as ci, but no more frequently than once per hour. This slashes our ci build time but still lets us know within one hour if we break any unit tests.

  • Similarly, don't build documentation, obfuscate, build installers, sign assemblies with certificates, etc, and disable any build processes that copy the outputs to the drop folder. CI builds are there to tell you if you've broken the build asap, you don't care about generating useful binary outputs.

  • optimise the build in general - merge projects together, use multi-threaded builds, use several build agents so ci builds don't have to wait for other build types to complete. Only do full builds overnight so your build server is dedicated to ci while you are working. Keep source files tidy (delete unused code rather than just commenting it out, remove unused usings/includes etc.)

  • invest in better build server hardware. If you don't have a top spec machine, drop more RAM and an SSD into it for a cheap speed boost. Make sure your build server is dedicated to ci builds, and isn't being used for anything else that might slow it down. Make sure the network between the build server and tfs sever is gigabit. Ensure you don't have any anti-virus software running on the server, or at least that its scheduled-scans are run overnight and your build folders are in the real-time-scan exclusion lists.

  • use tfs check in policies to stop devs checking in if the ci build has failed, so that you stop and fix breakages immediately.

Jason Williams
  • 56,972
  • 11
  • 108
  • 137
  • I think dropping the tests from the CI 'checkin' build may cause issues further down the line. Although we are only starting to see benefits from unit tests at this point, I can easily imagine a developer checking in code, breaking some test (and going home), while the hourly process catches it too late, making the nightly "big" build fail, etc. – lysergic-acid Dec 27 '11 at 12:05
  • All your unit tests should have passed before you checked in your code, otherwise how do you know it's safe to check in? And if one nightly build fails, is it the end of the world? A single failed build should not be a problem and you should plan to take it in your stride. – Jason Williams Dec 27 '11 at 18:50
  • I prefer having the build machine let me know if my tests passed. Do u run all tests manually beforehand? – lysergic-acid Dec 27 '11 at 22:23
  • @Jason It is often the case that unit tests could pass locally and still fail on build agents as they usually are different setups – Siy Williams Dec 28 '11 at 00:52
  • FYI tfs checkin policy you are referring to is "Gated Checkin" – The Muffin Man Feb 17 '14 at 21:23
  • Not quite... gated check-in is a build-with-shelveset, where the shelveset is automatically checked in IF the build succeeds. However, there is also a checkin policy that will not let you check in while any of the server builds are failed, which is what I was referring to. However, you're right that gated builds are now available, and provide a better way of handling the situation. – Jason Williams Feb 18 '14 at 20:19
11

I'm working on 500+ projects C# application. Projects are compiled in parallel and copylocal set to false. Compile time is about 37min without unit tests and code coverage. 13min for incremental build without any change in the code. If I turn off parallel compilation and set copylocal to true, compile time is > 1h40min. I have different configuration for local build, gated check-in build and server builds with deploy phase (night builds).

Here are my experiences:

  1. Copying output files to one directory is not good idea if you want to build your projects in parallel without CopyLocal set to false. My assemblies were sometimes locked when multiple projects referenced the same assembly and MSBuild tried to copy this reference to the output folder at the same time. This solution was very helpful for me. I set copylocal to false for all references and my build dir size was lowered 10x (10 times less I/O). I have different setup for local build and for server build. Different setup for gated check-in build and for full deploy build.
  2. If I enable parallel build, builds are faster, much faster. If you have strong build server your /m:2 build should be 2x faster as /m:1 build. It has nothing to do with dependencies between projects (if copylocal is set to false).
  3. You should reduce dependencies between the projects if you want to have fast incremental build. It has no impact on full build (copylocal false). Incremental compile time depends on the changed project location in the build tree.

Yes, MSBuild uses a timestamp of dependend projects to determine if a project needs a rebuild. It compares input files (code files, referenced assemblies, temporary files,..) timestamp with output assembly. If something is changed, your project is recompiled. Try to reduce number of depedencies between projects to minimize recompilation. If your change was only in the 'private' part of the project, your output assembly will be changed, assembly timestamp will be changed and all related projects will be rebuild also. You cannot do much with this.

Run your build 2 times with diagnostic verbosity without any change in your code and check for "Building target "CoreCompile" completely" like I described here. You can have something wrong in your project files and your projects are recompiled every time. If you don't change anything your build log should not contain "Building target "CoreCompile" completely" logs.

Our build server is virtual machine, not real piece of hardware. It is not good idea to use VM for build server, but it was not my decision.

If you have multi GB RAM try to use part of it as a in-memory hard drive. Your build should be much faster :)

SSD drives are sensitive to high I/O per day. It have an impact on warranty.

Hope it helps someone ... ;)

Community
  • 1
  • 1
Ludwo
  • 6,043
  • 4
  • 32
  • 48
6

The nightly build is less susceptible to slow build times so you should concentrate on optimising the checkin-triggered builds. We came across the same problem and found the following to help:

  1. Only building what you have changed. This means splitting out your projects into smaller ones.
  2. Build the solution in either Debug or Release (not both). Let the nightly build in both.
  3. Keep unit tests small and quick as to give fast feedback of results to the developers.
  4. Keep non critical tasks to the nightly build only (documentation, longer automation tests etc).
  5. Chain all dependent configurations so when your build succeeds they will build with the recent changes; if dependant configs fail then you know straight away that the changes have not integrating with existing code.

We tried Parallel MSBuild but I can't remember if it offered us much more; might be because of the agents we had.

Also, the checkout/checkin times had a huge impact on total time so maybe playing with the VCS so it only checks out changed files would help.

Siy Williams
  • 2,396
  • 1
  • 17
  • 26
  • Can you elaborate on point #5, and on the VCS tweak? I did not fully understand #5; Also how can i affect checkout times of my VCS ? (Git) – lysergic-acid Dec 26 '11 at 11:45
  • 1
    You can decrease checkout time in git using anent-side checkout and `teamcity.git.use.local.mirrors` option. If this option is set to true TeamCity creates a local mirror of your repository on each agent, updates it on every build (which is fast, since update is incremental) and clones from this local mirror, which is also fast since it is done on a single machine. Also in TeamCity 7.0 there is an option to do a shallow clone in git, which also decreases checkout time. – neverov Dec 27 '11 at 08:43
4

  1. Set Copy Local to false of course can save you tons of seconds.

  2. Besides, using RAM disk is also a good idea,

  3. Reduce your project number (merge them if possible)

References:

http://blogs.microsoft.co.il/blogs/arik/archive/2011/05/17/speed-up-visual-studio-builds.aspx

http://www.simple-talk.com/content/print.aspx?article=1237

Lex Li
  • 60,503
  • 9
  • 116
  • 147
  • I cannot set Copy Local = false since i would like to keep the project files as is (much cleaner). There is some way to override that when using MSBuild. – lysergic-acid Dec 26 '11 at 12:43
  • Regarding RAM Disk - not sure it would give better results, our CI server is a VM (VMware). Lastly, reducing the project number -- We are currently building a .SLN file, not sure how i can do it without major re-architecturing of our repository (which should be done anyway, but takes more time to complete). – lysergic-acid Dec 26 '11 at 12:44