4

I am trying to find a tool that can solve the following problem:

Our entire test suite takes hours to runs which often makes it difficult or at least very time consuming to find out which commit broke a specific test since there may be 50 to 200 commits in between test runs. At any given time there are only very few broken tests, so rerunning only the broken tests is very fast compared to running the entire test suite.

Is there a tool, e.g. a continuous integration server, that can rerun failed tests with a couple of revisions in between the last revision where the test was OK and the first revision where the test was not OK and therefore automatically find out on which specific commit a test switched from successful to broken.

For example:

Test A and B are ok in revision 100. Test A and B are broken in revision 200.

The tool should now run both tests with revision 150. Then if e.g. test A was broken and test B was OK in revision 150, it could continue to check test A with revision 125 and test B with revision 175 and so on until every broken test can be explained by some specific commit.

For a single test, I could probably hack something together with git bisect. But for multiple failed test this is probably not sufficient, since we need to search in both directions for many revisions.

intel4004
  • 41
  • 2
  • Could you not search for each test one at a time? Search and find where test A broke, then search and find where test B broke. – Shahbaz Apr 06 '12 at 13:04
  • Sure, it would be possible to only look at one test at a time. The problem is, that there are high setup costs for stuff that must be executed before a single test can be run (e.g. 5 min build followed by 1 second test). If 10 tests failed, checking a single revision would take 5 min + 10 * 1s with one approach and 10 * (5 min + 1s) with the other. – intel4004 Apr 06 '12 at 14:39
  • I don't know your project, but I assume you can build once and test many times, don't you think? Anyway, I'm not insisting! – Shahbaz Apr 06 '12 at 20:04

1 Answers1

4

git

Are you using or (see: What is Mercurial bisect good for?)?

Suppose version 2.6.18 of your project worked, but the version at "master" crashes. Sometimes the best way to find the cause of such a regression is to perform a brute-force search through the project's history to find the particular commit that caused the problem. The git bisect command can help you do this:[...]

From: Finding Issues - Git Bisect.

Essentially you mark two revisions as start and end and hit:

$ git bisect

Then run the test and depending on whether it passed or failled call

$ git bisect good

or 

$ git bisect bad

Respectively. git will do binary search, always cutting remaining revisions in half. I guess you can script it easily. If you are using , git can easily import the whole repository.

Building single revision at a time

This is not an answer but an advice - just test every single commit! Today you can cluster continuous integration servers easily, with a farm of servers it is not that hard to test all commits.

Community
  • 1
  • 1
Tomasz Nurkiewicz
  • 334,321
  • 69
  • 703
  • 674
  • Thank you, as I mentioned in my question, git bisect is definitely an option I am already considering. What I am trying to find out, is whether someone already solved this efficiently and without throwing tons of hardware at the problem. Yes, feedback would be quicker, if all test would start running immediately for every commit. Unfortunately this is currently unpractical since we have way to many long running tests for this. – intel4004 Apr 06 '12 at 14:51
  • We're using svn, but even though git-svn exists, it runs for about 4 hours before finally running out of memory when trying to import the repository. Whether this contradicts the claim "git can easily import the whole repository", I will leave to the jury to decide. To me, though, it sort of seems to be sucking at that. – Hakanai Apr 15 '15 at 01:17