who wrote 250k tests for webkit?

Question

assuming a yield of 3 per hour, that's 83000 hours. 8 hours a day makes 10,500 days, divide by thirty to get 342 mythical man months. I call them mythical because writing 125 tests per person per week is unreal.

can any wise soul out there on SO shed some light on what sort of mythical men write unreal quantities of tests for large software projects? thank you.

update chrisw thinks there are only 20k tests (check out his explanation below).
PS I'd really like to hear from folks who have worked on projects with large test bases

Answer 1

The original directory contained 240K files:

 Total Files Listed:
       243541 File(s)  1,062,470,729 bytes
       64718 Dir(s)

Many of these are svn files. If I remove all the subdirectories named ".svn" then the number of files drops to 90K:

 Total Files Listed:
       90615 File(s)    537,457,618 bytes
        7190 Dir(s) 

Some directories have a subdirectory named "resources" and/or "script-tests". I think that these subdirectories contain helper files which are used by test cases in the superdirectories. If I remove these subdirectories (because they don't add to the total number of tests) then the number of files drops to 87K :

 Total Files Listed:
       87672 File(s)    534,598,610 bytes
        6305 Dir(s)

Condensing 'similar' filenames (e.g. "arrow-keys-on-body.html" and "arrow-keys-on-body-expected.txt" are two files which define a single test) reduces the total number from 87K to 43K.

The only subdirectories which contain more than 1500 of these test cases (counted as described above) are:

2761   LayoutTests\dom
10330  LayoutTests\fast (of which 5934 are in LayoutTests\fast\js)
22575  LayoutTests\platform (with various O/S-specific subdirectories).

Within the platform subdirectories there seems to have been some copy-and-pasting between platforms. For example, the css3-modsel-37-expected.txt file exists:

• In the LayoutTests\platform\mac\css3 subdirectory
• In the LayoutTests\platform\chromium-win\css3 subdirectory
• In the LayoutTests\platform\qt\css3 subdirectory.

If I discard filenames which are duplicated into several platform subdirectories, then there are only 5716 (instead of 22575) unique platform tests.

In summary I think there are about 18K unique tests: which is still an impressive number of tests, but fewer than the 250K that you had estimated in your OP.

---

By way of comparison, I recently found CSS2.1 Test Suite: that looks like about 9000 test cases for CSS.

Answer 2

I just wrote 4 unit tests in 5 minutes. Not all unit tests takes long to make. Some are very simple ;)

Answer 3

• There are such things as automated unit test generators. For a complex function, you might often need to run all permutations of possible values of, say, 7 parameters. If they are boolean, you get 2⁷=128 tests. For certain scenarios, these 128 tests are all generated using 10 lines of code.

• Many regression-based unit tests can be generated by taking a large set of inputs, running the existing code on them, recording the corresponding outputs, and then making the gazillion tests which take new code, run it on same input and test that output matches old output.

• Writing unit tests is a fairly distributed endeavor. Armies of interns/volunteers can do it in parallel.

Answer 4

From the webkit contribution guidelines

"For any feature that affects the layout engine, a new regression test must be constructed. If you provide a patch that fixes a bug, that patch should also include the addition of a regression test that would fail without the patch and succeed with the patch. If no regression test is provided, the reviewer will ask you to revise the patch, so you can save time by constructing the test up front and making sure it's attached to the bug. If no layout test can be (or needs to be) constructed for the fix, you must explain why a new test isn't necessary to the reviewer."

Q) Who wrote those tests? A) Pretty much everyone who contributed to webkit.

Answer 5

These don't appear to be unit tests: they look more like system/regression tests.

This doesn't directly answer your question, but I've developed a browser of my own, so I might have have a little bit of insight into it; see:

• Should one test internal implementation, or only test public behaviour?
• Testing

can any wise soul out there on SO shed some light on what sort of mythical men write unreal quantities of tests for large software projects?

It's easy and necessary to write this kind of test:

• Easy: write an html page (or similar black-box input), to exercise functionality/features/behaviour
• Necessary:
  1. Need to test/exercise functionality when it's written
  2. Need to demonstrate/reproduce bugs when filing any bug report
  3. Need to keep all previous tests from 1. and 2., in order to do regression testing

A boss of mine once said, "You get what you inspect." (which implies that whatever you don't test is unknown/random).

Answer 6

It really depends on how this was counted.

But anyways, once a good framework for testing a specific class/library/unit/whatever is set up, adding a new test is usually only a few lines. If they were writing tests with a goal of high code coverage percentages then those tests are typically even shorter because you do a lot of tests that vary only by the value of a single variable in order to get a different code path.

Answer 7

Your analisys is incomplete, at best. Or is biased, I guess.

You should count the number of testers, some (or all) of them fully dedicated to test and only test, what make them more productive. Also, you should realized that very few tests are complex in lots of projects and most of tests could be repetitive, so they could be easily implemented.

Answer 8

In comments, you asked:

who watches the watchers? somebody must have the nightmarish job of herding all these ducklings

... and ...

right, you're asking the same question I am. how many testers were there. who are they. are they developers or QA staff? volunteers?

Perhaps your questions are answered at WebKit Committers and Reviewer Policy.

Answer 9

assuming a yield of 3 per hour, that's 83000 hours

Given that there are only 18K unique tests, and 200 working days per person-year, then if you budget an entire day to develop each test, then a team of 9 full-time testers/people could develop 18K tests in 10 years (the KDE's KHTML and KJS projects began in 1998). These numbers are probably irrelevent (it may not take day to develop each new test case, and they may not have full-time/dedicated testers), but they do IMO illustrate that 18K tests over 10 years is feasible for a 'large' (or non-trivial), successful project.

I'm using a similar test strategy in my project: not because I copied Webkit, but because it's the only way I think of to do automated-but-maintainable regression tests of a GUI.

The following FYI is an example (one of the shortest/simplest examples) of one of my test cases. I generate it (and other like it) simply by manually exercising my browser's UI within a built-in test-case-generation framework. I can then run again later as an automated regression test (the framework replays the input and asserts that the output hasn't changed).

I find that creating the test cases is not what takes a relatively long time.

> clientSize 20 10
+ screenDisplays lines 0
----------
----------
> loadDocument lines 5
----------
<div>
<h1>Easy title</h1>
<p>Hello world. Lorem ipsum.</p>
<p>Embedded <a>anchor</a> tag.</p>
</div>
----------
< invalidateAll
> paint 0 0 20 10
+ screenDisplays lines 10
----------
····················
····················
··»«Easy title········
····················
····················
··Hello world. ·····
··Lorem ipsum.······
····················
····················
··Embedded anchor ··
----------
> mouse down 9 2
< ensureVisible 9 2 1 13
+ screenDisplays lines 10
----------
····················
····················
··Easy ti»«tle········
····················
····················
··Hello world. ·····
··Lorem ipsum.······
····················
····················
··Embedded anchor ··
----------
> mouse up 9 2
< ensureVisible 9 2 1 13
+ screenDisplays lines 10
----------
····················
····················
··Easy ti»«tle········
····················
····················
··Hello world. ·····
··Lorem ipsum.······
····················
····················
··Embedded anchor ··
----------
> keyDown Enter
< invalidate 2 2 16 1
< invalidateAll
< invalidate 2 4 16 3
< ensureVisible 2 5 1 16
+ currentDocument lines 6
----------
<div id="ModelText_id_contents">
 <h1>Easy ti</h1>
 <h1>tle</h1>
 <p>Hello world. Lorem ipsum.</p>
 <p>Embedded <a>anchor</a> tag.</p>
</div>
----------
+ debugDumpDom lines 15
----------
 1  div
 2    h1
 3      Easy ti
 4    h1
 5      tle
 6    p
 7      Hello world. Lorem ipsum.
 8    p
 9      Embedded 
10      a
11        anchor
12       tag.

Selected start={parentId=4,parent={tle},offset=0}, end={parentId=4,parent={tle},offset=0}

----------
> paint 0 0 20 10
+ screenDisplays lines 10
----------
····················
····················
··Easy ti···········
····················
····················
··»«tle···············
····················
····················
··Hello world. ·····
··Lorem ipsum.······
----------