Using JUnit as an acceptance test framework

Question

OK, so I work for a company who has openly adopted agile practices for development in recent years. Our unit tests and code quality are improving. One area we still are working on is to find what works best for us in the automated acceptance test arena. We want to take our well formed user stories and use these to drive the code in a test driven manner. This will also give us acceptance level tests for each user story which we can then automate.

To date, we've tried Fit, Fitnesse and Selenium. Each have their advantages, but we've also had real issues with them as well. With Fit and Fitnesse, we can't help but feel they overcomplicate things and we've had many technical issues using them. The business haven't fully bought in these tools and aren't particularly keen on maintaining the scripts all the time (and aren't big fans of the table style). Selenium is really good, but slow and relies on real time data and resources.

One approach we are now considering is the use of the JUnit framework to provide similiar functionality. Rather than testing just a small unit of work using JUnit, why not use it to write a test (using the JUnit framework) to cover an acceptance level swath of the application? I.e. take a new story ("As a user I would like to see basic details of my policy...") and write a test in JUnit which starts executing application code at the point of entry for the policy details link but covers all code and logic down to the stubbed data access layer and back to the point of forwarding to the next page in the application, asserting on what data the user should see on that page.

This seems to me to have the following advantages:

• Simplicity (no additional frameworks required)
• Zero effort to integrate with our Continuous Integration build server (since it already handles our JUnit tests)
• Full skillset already present in the team (its just a JUnit test after all)

And the downsides being:

• Less customer involvement (though they are heavily involved in writing the user stories in the first place from which the acceptance tests will be written)
• Perhaps more difficult to understand (or make understood) the user story and acceptance criteria in a JUnit class verses a freetext specification ala Fit or Fitnesse

So, my question is really, have you ever tried this method? Ever considered it? What are your thoughts? What do you like and dislike about this approach? Finally, please only mention alternative frameworks if you can say why you like or dislike them more than this approach.

Answer 1

My experience using JUnit for an Acceptance Testing framework.

Our company did a very similar thing where we started with FitNesse and then went to JUnit. We did this mainly because FitNesse was sitting on top of Java already, so switching to JUnit was easy. FitNesse seemed to not fit (no pun intended) our needs.

Here are my pros and cons of using JUnit:

Pros:

• JUnit has a lot of flexibility (because it's Java), so it's relatively easy to create an Internal Domain Specific Language[Fowler]. The DSL can be modified so that it's easy for domain experts (not software engineers) to read/write.
• Since it's JUnit you can use Eclipse as an IDE, which gives you auto-complete, syntax checking, debugging, etc.
• Specific for our application, we have a CORBA interface to the UI. Java can easily use this CORBA interface as well.

Cons:

• People hear JUnit and think unit test. This gets confusing when people start referring to the acceptance tests as "JUnit tests".
• To provide a simplified testing environment you have to spend some time extending the framework and developing your DSL.

So in summary, JUnit works just fine for Acceptance Tests if you are willing to spend the time to extend it into you own domain specific testing environment. Otherwise I think I would look elsewhere for something more out of the box.

Answer 2

Business and UAT
Business most of the time will loose interest in any testing tool sooner or later. They are Business not testers after all, so they won't commit much to writing/maintaining test scripts. They are Business, they want to do business. From their point of view testers/developers/other IT guys are responsible for delivering them software of certain quality.
Even if Agile is your way, and you get some level of commitment from business don't expect them to behave like tester with every sprint/iteration or whatever you have. It is really not their job.
One digression: User Acceptance Tests are be manual tests executed by the user(s), so he(they) can say is product what he(they) asked for.
So whenever feature (or set of features) is ready, do presentation for business, let them play with it, let them say is it what they need. Don't force them to check this feature with every iteration.

Now back to your acceptance tests. Whenever you have story from client to do, implement test(s) for it in your test infrastructure. You write it, you run it, you maintain it. It may be wise to have test for feature and drive development with ADD/BDD/TDD or whatever you want to call it. It should be obvious that this test(s) than contribute to your regression tests suite in next iterations.

So my point of view is, Business won't be keen on writing/maintaining tests, especially when set of features grows. Don't fight with it, adapt to it. Let them do just (manual) User Acceptance Tests at the end of iteration for new features. For your own sake create tests for features they want have. Create feature acceptance tests for each feature. Let those tests become your Regression Testing Suit. Accept that t is your responsibility to maintain it.

Acceptance testing Framework
JUnit, huh? Then try with WatiJ for web, and Abbot for desktop. Keep in mind that those test won't be simple Unit Tests. You want something that will tests features of real application, you want Test Script to execute specific Scenario/Business Process. This means you need to approach writing those tests like writing application they are testing: DRY,KISS, YAGNI, Design Patterns - everything applies. In the end, it will be writing software, that just happens to test other software you write.

Will those test become large and complex with this approach? Well, larger and more complicated than unit tests. But they are not testing single unit but whole application. Besides they will be far more simpler and smaller than application you are writing in the first place.

Will you bee writing testing framework? If you want to succeed with this approach, than yes, you will be writing testing framework - collection of test classes and helper classes that have some knowledge about your application implemented. But you won't be extending JUnit. JUnit here is just some API you use in your framework.

Answer 3

It is perfectly valid to use JUnit to write acceptance tests. JUnit is just a test execution framework, it is completely irrelevant whether you are actually writing unit tests with it, component tests, integration tests or acceptance tests. As user1704737 already said it is important that you have a readable DSL for writing your tests.

To write you tests in JUnit I suggest to give JGiven a try. It is a lightweight framework to make it easy to write acceptance tests in the given-when-then notation in plain Java. It uses JUnit (or TestNG) as a test executor, so that it can easily be integrated into existing test infrastructures. You also get HTML reports for your acceptance tests which you can show business to validate the tests against the requirements.

Disclaimer: I am the author of JGiven

Answer 4

Try robotframework (www.robotframework.org), I've been using it for years, Pekka Klarck from Helsinki developed it (Nokia Siemens Networks is supporting it since the beginning, now it's also open sourced).

robotframework uses tabular format test cases, supporting HTML, CSV or plain text (written in certain grammer). It has a library mechanism, which supports to import Selenium functions. You can write library for robotframework in Python or Java.

For the JUnit as Acceptance Test Framework, of course you can, but it doesn't worth doing. Higher level testing should rely much less even none on a software's internal implementation, while write Acceptance Test using JUnit (which is a unit test framework), you introduced a lot dependencies on its e.g. internal function calls. Which will be a big headache while there is any changes with the software design, which a high level test should not be affected and you're.

Answer 5

A few recommendations if you follow that path:

• Put your acceptance tests in another project, since probably they will run slower and have more dependencies than your usual tests
• Create builders (maybe using a fluent interface) to setup your model in test fixtures, because you will find that setting up the required objects for your test will be the most boring and difficult part to maintain.
• Take a look into Groovy, and the Spock Framework: it's compatible with JUnit, but provides you with a nice DSL to make your tests readable.

Answer 6

You mentioned some good points about the pros and cons. Can I add:

Upsides: I like the idea of being able to test everything without having to have a database, a web server. This helps enormously in understanding the code, and in refactoring.

Downsides:

The JUnit code will be complex and hard to maintain. You will have to reproduce all of the interactions between code and user within the Junit test, which very quickly becomes complex. If all you are doing is testing one single entry point, then this could well work, but if you are testing multiple screens (a wizard or something like that), then it can quickly become fragile and unmanageable. The problem in my experience is almost always the plumbing code between pages, or the setup code required for a page.

One other thing to consider is the flexibility of what you're trying to do. With Fitnesse, it is fairly easy to add another test if you discover a bug. In fact, this is the idea behind Fitnesse: the 'user' can add another test without having to change the code. Don't underestimate the value of this.

So I would suggest two things:

1) Try it. Take a user story (not too complex, not too simple), and do it in JUnit. Compare what you've done with the Selenium/Fitnesse etc, and see if it is worthwhile. See if you 1. actually find bugs, 2. produce brittle, hard to manage code.

2) Could you use the data created by the Fitnesse as input to your JUnit tests (thus removing the need for a web server etc). You could possibly read the data and call the methods in the Java classes directly. Again, you may have problems with setup & database code.

Answer 7

IMHO Not a good idea. Apart from the 2 cons that you mentioned, how about the vast amount of effort required to build an acceptance test framework that will deal all kinds of web/UI/glue scenarios that you might have? Your point "Simplicity (no additional frameworks required)" is incorrect as your organization would have to invest on building a more elaborate framework on top of JUnit. It may be more than JUnit.