2

We are moving to an Acceptance Test Driven Development approach for defining features. It seems to be working well, but we're starting to run into issues with test management. At the moment, we use SharePoint/Excel to track the acceptance tests. This is because non-technical customers, QA and dev all update the tests. The problem is that the tests don't live with the code, so they aren't branched/versioned along with the code, and it is all very manual. We're looking at full on test case management software (e.g., Zephyr, TestRail, etc), but that feels a little heavy, and ultimately the test data still doesn't live with the code.

Is there a test management application that is friendly to non-devs, but stores data in a way that will work with git? Is trying to keep the tests alongside the code a fool's errand?

Thanks, Erick

user3632158
  • 277
  • 1
  • 6
Erick T
  • 7,009
  • 9
  • 50
  • 85

2 Answers2

1

Yes, I like your idea to keep the tests together with the code.

One framework that you could use to achieve this goal is FitNesse. It is a well known test management tool that supports collaboration between business guys, developers, and testers. With the Git Plugin for FitNesse you can keep your tests within a Git repository.

We maintain our acceptance tests, which are based on another framework named Concordion, also in our source code repository. Concordion specifications consists of two parts: a well-formed HTML document describing the functionality, and fixture code written the programming language of your application such as Java or C# that finds concrete examples in the document and uses them to validate the system under test.

Our executable specifications are written with a WYSIWYG HTML editor such as Microsoft WebExpression or BlueGriffon by the product owner or testers. We keep them in our repository and access them via TortoiseGit, which works very well as all members of our team have technical background. To involve non-technical you would probably need to write a small script that pulls the latest updates into the local repository, start the editor of your choice, and push the changes after editing back into the central repository.

Please, note that there is a Concordion extension for Excel available, which enables to specify your test data (i.e. the input and expected output values) in an Excel spread sheet and use this to test your application.

IMHO testing is often some sort of detailed analysis in software projects. As many details are not discussed and specified, testers do often the job of thinking about the details of a feature. Unfortunately, this happens after developers did their work, which causes a lot of rework and additional cost. Thus, base on your ATDD effort you should try to improve the collaboration of your team on specifications before you head for test automation.

Please, have a look at a talk of Gojko about the benefits to invest 5-10% of a sprint to define details collaboratively. The specification work in our team has improved a lot since we do specification workshops based on the ideas of specification-by-example.

user3632158
  • 277
  • 1
  • 6
  • Do you have any suggestions as to how to incorporate manual testing? What we are struggling with right now is how to maintain a list of tests, some of which are automated, some of which are manual. How to track the "execution" of manual tests, for example. – Erick T Oct 27 '14 at 23:22
  • 1
    You can define in your team the edge cases for your features or user stories, and discuss concrete examples. When you use tools with a [business readable input]( http://blog.jonasbandi.net/2010/03/classifying-bdd-tools-unit-test-driven.html) you can write specifications both for manual and automated tests. Please, see an example for [a specification based on Concordion](http://concordion.org/Example.html). When the feature has to be tested, the involved tester makes sure that all tests have been executed (by reviewing the automated tests and doing the manual testing). – user3632158 Oct 29 '14 at 10:44
0

I think storing tests alongside the code is an excellent idea.

Any framework that uses plaintext files as assets can be stored alongside the code. For example, robot framework lets you create tests in plain text format.

These frameworks aren't full on test environments, however, but with a little education anybody can maintain the test cases

Bryan Oakley
  • 370,779
  • 53
  • 539
  • 685
  • 1
    The team has some concern about non-technical users running free in git. Knowledge of how to use git is not easy to acquire for non-technical people, and it's easy to shoot your foot (and thus dev's feet) off. GitHub for Windows might be a solution. – Erick T Oct 31 '14 at 15:16