16

We have the "problem" of a large automated suite of integration tests. While our build times are reasonable (< 1 hour), the tests typically take > 6 hours to complete.

While it's great to have this large chunk of functionality tested in our build runs, it obviously is a barrier to implementing CI, which I've found to be a very helpful for keeping source trees in a "always buildable" state.

I've reviewed threads of discussion like this one, which elaborate on the distinctions.

This leads me to a few questions:

  1. Does CI dictate or recommend Unit vs. Integration testing automation? I've heard Unit-only in the past, but am not finding any such statements (or rationale) for this in a quick search.

  2. What is a good "best practice" for combined build + automated test times/ratios to have effective CI for a team? My gut tells me that this should be < 2 hours as a worst case, and probably < 1 hour to be really effective. In theory, we could break up the tests to run in parallel and probably get them running in under 2 hours, but this would still be a 3 hour run.

  3. What's the best way forward from long-running Nightly Builds + Integration Tests to CI? I'm thinking of a CI build with a few skeletal Unit Tests only, in combination with nightly builds that continue with the integration tests.

Any tooling recommendations are also welcome (Windows-only C#/C++ codebase)

Community
  • 1
  • 1
holtavolt
  • 4,378
  • 1
  • 26
  • 40
  • Update - items 1-3 addressed, but didn't receive any tooling recommendations. CruiseControl.NET is the obvious pick - any others worth considering for a C#/C++ Windows-only codebase? – holtavolt Feb 04 '11 at 16:48
  • Just stumbled on this. We are trying out [Jenkins](http://jenkins-ci.org/) for Windows C# and got pretty far in under a day. Also look at TeamCity and Bamboo – KCD Jun 26 '12 at 21:57

2 Answers2

14

For most projects, however, the XP guideline of a ten minute build is perfectly within reason. Most of our modern projects achieve this. It's worth putting in concentrated effort to make it happen, because every minute you reduce off the build time is a minute saved for each developer every time they commit. Since CI demands frequent commits, this adds up to a lot of time.

Source: http://martinfowler.com/articles/continuousIntegration.html#KeepTheBuildFast

Why does it takes 6 hours? How many tests do you have? What are the ratio of the unit-test compared to integrated ones? You probrably have many more integrated tests or your unit-test are not really unit. Are your unit tests touching the DB? This may be the problem.

6 hours is a long long time. The article above has some tips.

goenning
  • 6,514
  • 1
  • 35
  • 42
  • Thanks for the link to the Fowler article. Yes - our testbed is almost entirely integrated tests, which is clearly part of the problem (and something I'd like to convince the team and management is worth investing in changing) – holtavolt Feb 04 '11 at 15:52
  • 3
    One thing to consider is test prioritization. Could you at least convince the team to break up the tests into Priority 1, 2, and 3 suites and just run the Priority 1 tests in the CI? Of course this would mean that the test team would have to periodically (daily?) run the rest of the tests on a given blessed build to ensure proper coverage. As a guideline, Pri1 tests could be a small set of the key scenarios/integration tests that, when fail, means that the build is generally toast and major functionality isn't available, for example. – nithins Feb 04 '11 at 15:58
  • if a test isn't run in CI then when it breaks nobody will know. so if you don't run it, then delete it, at least then you are being honest with yourself. – time4tea Feb 04 '11 at 16:00
  • Fowler's pointer to TestDouble test servers is interesting. I think nithins point on prioritization tiers of tests will help, too. I think P1 should be primarily Unit Tests (most of which will have to be written) and a few of the key (and fast) integration tests. – holtavolt Feb 04 '11 at 16:21
  • 1
    Actually, this is called `smoke testing`. They run before any other test and stop the builds if it fails. This is exactly what @nithins said. Just google for `smoke testing` and you'll find some references about it. – goenning Feb 04 '11 at 17:39
  • The term "smoke test" is already in use here to indicate an initial manual testing procedure. Reviewing the Wikipedia entry: http://en.wikipedia.org/wiki/Smoke_testing#Software_development , it looks like "build verification test" would be a bit more precise (and non-overloaded in our org) – holtavolt Feb 04 '11 at 18:07
5

There are a few things here.

In general you will have a number of builds, one that compiles & runs unit tests, one that does that and runs local acceptance tests, and one that runs integration tests.

You definately don't need a single build that does everything.

Your build times to me sound pretty long - remember that the point here is to give quick feedback that something has gone awry. I don't know much about your project - but i would think that you should look to get your compile and unit test build down to under two to three minutes. This is perfectly achievable, in all but very large projects, so if your unit tests take along time, then its time to start asking why.

6 hours is also a very long time. are you sure that your tests are testing the right stuff? do you have too many wide scope tests? are you using "sleep()" everywhere to makeup for the fact that you haven't modeled asynchrony well in your test code?

You should probably get hold of Jez Humbles book "Continuous Delivery", and take a look at Growing Object Oriented Software as how to write unit / integration tests. GOOS uses Java as an implementation language, but all the concepts are the same.

time4tea
  • 2,169
  • 3
  • 16
  • 21
  • by the way - unit tests are supposed to be quick! - you should be able to run several thousand unit tests in under a minute. – time4tea Feb 04 '11 at 15:40
  • Yes - I understand these are the intentions of CI, and I will emphasize that we are running *integration tests*, and not Unit Tests currently. These integration tests are extensive, and as bugs are found and fixed new tests are added to cover these cases (e.g. regression tests mixed in). – holtavolt Feb 04 '11 at 15:50
  • 1
    Well, sure - but how many do you have? what stuff do they cover? how much duplication is there? are there sleep()'s all over the place? why do they take so long? what do you mean bugs? do testers just report a bug if thy find something in UAT that doesn't work? thats not a bug, just an unfinished feature, so write a lower level test.... – time4tea Feb 04 '11 at 15:56
  • you also ask how long builds should be to be "really effective". well they should be short. integration builds maybe 10 mins max, comile and unit test about 30 sec ideally – time4tea Feb 04 '11 at 15:57
  • Splitting up the testsuites enables a flexible policy. We have the policy that code **must** pass unit tests before a check-in, but need not pass integration tests. – Raedwald Feb 04 '11 at 16:20
  • I'm sure that a CI build that does incremental updates/builds will be fast (on the order of minutes). Our current builds are "clean slate", so that's not an issue. Regarding the number and coverage, they are reasonable for the application suite (plenty of DB and network file activity) - no big gains to be had there. – holtavolt Feb 04 '11 at 16:24