12

I am using Selenium to test my java web app's html pages (JSPs actually). My web app requires a flow to access each pages (it is a small online game web app), as in: to get to page B you'd need to go to page A, enter some text and press a button to get to page B. Obviously I already have some tests to verify that page A is working properly.

I would like to be able to write more tests in order to check that after the tests for page A run I'll get my tests for page B running (and so on for the rest of the app). So in short: define some order in my tests somehow.

After doing lots of reading about testing in the last few days, I cannot find anything interesting on this specific subject. Hence I'm asking for advice now.

Possible Solutions I have identified:

  1. Define (in the same test class) test methods for page A, then test methods for test B. Then order the execution of the test methods. But we know JUnit (but TestNG does) does not allow test methods execution ordering, see SO question selenium-junit-tests-how-do-i-run-tests-within-a-test-in-sequential-order

  2. Grouping all the tests (for page A, page B, & so on) under one test method. But I've read it's bad, see SO question: junit-one-test-case-per-method-or-multiple-test-cases-per-method. Is it that bad when doing selenium test? I've seen some code doing it so I assume it may not be.

  3. Grouping all the tests (for page A, page B, & so on) under one test method but use the JUnit's ErrorCollector class: ErrorCollector allows you to execute ordered checks in the same method and yields a specific error message if one fails BUT let the method (hence the checks) running until the end. This solution seems too "brutal" to me.

  4. Use JUnit's TestSuite class: it runs test listed in the suite in the order test classes are defined in the suite. So that would involve having independent test methods to test page A in a test class (let's say TestA), then all the test methods to test page B in a test class (let's say TestB), and so on. Then insert those in a test suite such as @SuiteClasses( {TestA.class, TestB.class, TestC.class, ...} )

  5. Use JUnit's TestSuite class in combination with JUnit's ErrorCollector class. Oh well, since we can, you may want to Group test per page in different classes, & on top of that group page tests "zones" using ErrorCollector. This solution may be very usefull if you have a very dense web page or other reasons.

  6. Quite radical: Use another tool such as TestNG to have access to features such as test method ordering.

Note: I imagine some would recommend the last solution (migrate to TestNG) but I'd like to hear other ideas/opinions tied to JUnit too. As in, if I'm working in a team that is not able (for some reason) to migrate to another testing framework, then how would they address this test ordering issue?

Community
  • 1
  • 1
Adriano
  • 19,463
  • 19
  • 103
  • 140
  • 1
    This might just be the best qualified, and well structured question I have seen on SO. – jumps4fun Feb 11 '15 at 11:13
  • Thank you. This is very flattering. I'm just trying my best, after learning from others on stackoverflow. I got to say, this website does teach much more than IT, the more I use it, the better I get at putting complex problems into written & clear words – Adriano Feb 11 '15 at 11:50
  • ..and more often than not, that actually solves the problem before I finish asking the question :). Have you started using one of the suggested tools for behaviour testing? I would love to hear it. – jumps4fun Feb 11 '15 at 12:11
  • Yes, that right. I didn't get much time to use those tools in the end, as I soon moved to a front end job. I did try TestNG and found it nice to use, more convenient/intuitive than JUnit in my biased opinion ;-) – Adriano Feb 11 '15 at 13:09

5 Answers5

8

Why to migrate? You can use JUnit for unit-testing and another framework for higher-level testing. In your case it is a kind of acceptance or functional or end-to-end, it is not that important how you name it. But important is to understand that these tests are not unit. They stick to different rules: they are more complex, run longer and less often, they require complex setup, external dependencies and may sporadically fail. Why not use another framework for them (or even another programming language)?

Possible variants are:

If adding another framework is not an option: you enumerated more options for JUnit then I could imagine =) I would put the whole test script for the flow in one test method and would organize test code into "Drivers". That means that your end-to-end tests do not call the methods of your application or Selenium API directly, but wrap them into methods of Driver components which hide API complexity and look like statements of what happens or what is expected. Look at the example:

@Test 
public void sniperWinsAnAuctionByBiddingHigher() throws Exception {
    auction.startSellingItem();

    application.startBiddingIn(auction);
    auction.hasReceivedJoinRequestFrom(ApplicationRunner.SNIPER_XMPP_ID);

    auction.reportPrice(1000, 98, "other bidder");
    application.hasShownSniperIsBidding(auction, 1000, 1098);

    auction.hasReceivedBid(1098, ApplicationRunner.SNIPER_XMPP_ID);

    auction.reportPrice(1098, 97, ApplicationRunner.SNIPER_XMPP_ID);
    application.hasShownSniperIsWinning(auction, 1098);

    auction.announceClosed();
    application.hasShownSniperHasWonAuction(auction, 1098);
} 

A snippet is taken from the "Growing Object-Oriented Software Guided by Tests". The book is really great and I highly recommend to read it.

This is real end-to-end test that uses real XMPP connection, Openfire jabber server and WindowLicker Swing GUI-testing framework. But all this stuff if offloaded to Driver components. And in your test you just see how different actors communicate. And it is ordered: after application started bidding we check that auction server received join request, then we instruct auction server to report new price and check that it is reflected in UI and so on. The whole code is available on github.

The example on github is complex, because the application is not as trivial as it usually happens with book examples. But that book gives it gradually and I was able to built the whole application from scratch following the book guide. In fact, it is the sole book I ever read on TDD and automated developer testing that gives such a thorough and complete example. And I've read quite a lot of them. But note, that Driver approach does not make your tests unit. It just allows you hide complexity. And it can (and should) be used with other frameworks too. They just give you additional possibilities to split your tests into sequential steps if you need; to write a user readable test cases; to externalize test data into CSV,Excel tables, XML files or database, to timeout your tests; to integrate with external systems, servlet and DI containers; to define and run separately test groups; to give more user-friendly reports and so on.

And about making all your tests unit. It is not possible for anything excluding something like utility libraries for math, string processing and so on. If you have application that is completely unit tested that it means either that you test not all application or you do not understand what tests are unit and what are not. The first case may be OK, but everything that is not covered must be tested and retested manually by developers, testers, users or whoever. It is quite common but it better to be conscious decision rather than casual one. Why you cannot unit test everything?

There are a lot of definitions of unit tests and it leads to holy war) I prefer the following: "Unit test is test for program unit in isolation". Some people say: "hey, unit is my application! I test login and it is simple unit function". But there is also pragmatics that hides in isolation. Why do we need to differ unit tests from others? Because it is our first safety net. They must be fast. You commit often (to git, for example) and you run them at least before each commit. But imagine, that "unit" tests takes 5 minutes to run. You will either run them less often or you will commit less often or you will run just one test case or even one test method at the time, or you will wait say each 2 minutes for tests to complete in 5 minutes. An in that 5 minutes you'll go to Coding Horror where you'll spend the next 2 hours =) And unit tests must never fail sporadically. If they do that - you will not trust them. Hence, the isolation: you must isolate slowness and sources of sporadic failures from your unit tests. Hence, isolation means that unit tests should not use:

  • File system
  • Network, Sockets, RMI and so on
  • GUI
  • Multithreading
  • External libraries expecting test framework and supporting simple libraries like Hamcrest

And unit tests must be local. You want to have just one or so tests failing when you've made a defect within 2 minutes of coding, not a half of the whole suite. That means that you are very limited in testing stateful behavior in unit tests. You should not make a test setup that makes 5 state transitions to reach preconditions. Because fail in first transition will break at least 4 tests for following transitions and one more test that you currently write for the 6th transition. And any non-trivial application has quite a lot of flows and state transitions in it. So this cannot be unit tested. For the same reason unit tests must not use changeable shared state in database, static fields, Spring context or whatever. This is exactly the reason why JUnit creates new instance of test class for every test method.

So, you see, you cannot fully unit test a web app, no matter how you recode it. Because it has flows, JSPs, servlet container and probably more. Of course, you can just ignore this definition, but it is damn useful) If you agree that distinguishing unit tests from other tests is useful, and this definition helps to achieve that then you'll go for another framework or at least another approach for tests that are not unit, you'll create separate suites for separate kinds of test and so on.

Hope, this will help)

Rorick
  • 8,857
  • 3
  • 32
  • 37
  • I get what you mean: what I'm trying to do is not Unit testing, hence JUnit is most likely not the best tool for this (hence its name). I'll look into the other frameworks you suggest. But I was thinking about another solution too: recode my web app so that the web pages can be accessed without "logic" dependencies between each others (prob a neater design too). Once this is done, then I could test pages in isolation hence be performing unit testing, am I correct? – Adriano Sep 13 '12 at 15:34
  • I did not get the "organize test code into Drivers" thing. Tried to by having a look at the github repo but this example is prob too hard to start with. Thx for the book recommendation, added to my ToRead list :) – Adriano Sep 13 '12 at 15:43
  • I will expand answer, because it is too long for comments, OK? =) – Rorick Sep 13 '12 at 18:46
  • Phew! That was long indeed. Thx for the effort, you do deserve the correct answer considering how much explanation you did. Ps: will defo read that book, but it'll have to wait..already got a few on my shelf! – Adriano Sep 13 '12 at 22:19
2

You can implement your own Runner that will probably wrap other Runner and sort tests according to criteria yo9u defined. This is if you really need this.

But I do not think it is so necessary. I understand that if page A does not work the pass to B will not work too. So you want to run the test A first and then run test A->B. But does it really make sense? If for example test A->B runs first and fails because it cannot arrive to page A other test that verifies test A will fail too. So, Both tests will fail and id does not depend on the tests order.

If however you mean that you want to use test A as a set up operation of test B it is very bad practice. You can use the logic of test A as a start of test B but you should not couple between 2 tests. One obvious reason is that this makes debugging very hard. To debug test A->B you have to run both A and A->B tests that means that you probably have to run all tests (at least within one test case).

AlexR
  • 114,158
  • 16
  • 130
  • 208
  • 1
    It is very bad practice for unit testing, but not for acceptance testing as the question supposes. In fact, *not* using previous steps as a setup could be a bad practice in acceptance. For instance, it can lead to very long-running tests which is always bad for both unit and acceptance. – Rorick Sep 13 '12 at 11:47
1

We use a Framework call Cucumber. Its behavioral driven development. So essentially, you can create testing function that are agnostic of the flow and use feature to control the flow of your testing.

Example Page 1, enter 2 input, click enter, verify page 2 loaded with a something.

Then you'll have this in a feature file:

Given Page 1
   Then I enter text1
   Then I enter text2
   Then I click button
   Then I see page 2

And in you code, you can have a class that implements these steps. And with the cucumber framework, you can use annotation to denote the mapping between you testing code and the feature file.

...
@Given("^Page 1$")
public void iLoadPage1() {
  WebDriver driver = new ....
  driver.go('URL');
}

@Given("I enter (.*)$")
public void iEnterTest(String txt) {
  ...
}

...
Churk
  • 4,556
  • 5
  • 22
  • 37
  • I think this is very similar to testNG which I haven't use yet. – Churk Sep 13 '12 at 11:16
  • No, TestNG does not give anything similar to BDD. It uses direct dependencies between test methods and/or groups. Literally, you state: "method A depends on method B"; and this guarantees that method A will be called after method B. And that's all. – Rorick Sep 13 '12 at 12:32
1

JUnit is not really designed for flow/integration level testing.

For that situation, I've designed my own Runner that guaranties the order the tests are run and also reuses the same instance of the test class for all test so you can pass values for a step to another.

(Yes, it's a bad practice for unit tests - but we're not talking about unit tests there even if they run with jUnit).

Using another tool (cucumber, fitnesse, TestNG, whatever) is also a good solution - but there is such thing as too many test tools in a project.

ptyx
  • 4,074
  • 1
  • 19
  • 21
1

Another option that may work is applying JUnit Parameterization to your test. It is my present understanding that the Parameters for the implementation are always executed in the order they are provided.

Using that concept you could have your JUnit implementation accept the URL as a constructor argument and fork the test internally based on the parameters provided.

In order to ensure that you're using the same WebDriver reference it would probably need to be static @BeforeClass/@AfterClass declarations. With that you may be able to have the parameters chain off one another, effectively testing that "From the previous test I am on page X. While here I will perform task Y. At the end of this test I will be on page Z, or in state A".

In Unit-level testing I would certainly say this solution would be bad form, but when you integrate a tool like Selenium you start acting on the integration-test level. I'm fairly new to this concept myself, but in the integration-test level it seems the rules of modularity are a bit more fuzzy since you will have conditions that are dependent.

I was curious, so I tried it out. It acts like I was thinking it would, if we assume we can treat the application as a static resource in relation to the test.

package demo.testing;

import java.util.List;

import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

import com.google.common.base.Predicate;
import com.google.common.base.Predicates;
import com.google.common.collect.Lists;

@RunWith(Parameterized.class)
public class SequentialParams {

    private static SystemState state;

    @BeforeClass
    public static void validateBeforeState() {
        state = new SystemState();

        Assert.assertFalse(state.one);
        Assert.assertFalse(state.two);
        Assert.assertFalse(state.three);
        Assert.assertFalse(state.four);
    }

    @Parameters
    public static Object buildParameters() {
        Runnable reset = new Runnable() {

            public void run() {
                state.one = false;
                state.two = false;
                state.three = false;
                state.four = false;
            }
        };
        Runnable oneToTrue = new Runnable() {

            public void run() {
                state.one = true;
            }
        };
        Runnable twoToTrue = new Runnable() {

            public void run() {
                state.two = true;
            }
        };
        Runnable threeToTrue = new Runnable() {

            public void run() {
                state.three = true;
            }
        };
        Runnable fourToTrue = new Runnable() {

            public void run() {
                state.four = true;

            }
        };


        Predicate<SystemState> oneIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.one;
            }
        };
        Predicate<SystemState> twoIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.two;
            }
        };
        Predicate<SystemState> threeIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.three;
            }
        };
        Predicate<SystemState> fourIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.four;
            }
        };

        Predicate<SystemState> oneIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.one;
            }
        };
        Predicate<SystemState> twoIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.two;
            }
        };
        Predicate<SystemState> threeIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.three;
            }
        };
        Predicate<SystemState> fourIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.four;
            }
        };
        List<Object[]> params = Lists.newArrayList();

        params.add(new Object[]{Predicates.and(oneIsFalse, twoIsFalse, threeIsFalse, fourIsFalse), oneToTrue, Predicates.and(oneIsTrue, twoIsFalse, threeIsFalse, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsFalse, threeIsFalse, fourIsFalse), twoToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsFalse, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsTrue, threeIsFalse, fourIsFalse), threeToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsFalse), fourToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsTrue)});

        params.add(new Object[]{ Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsTrue), reset, Predicates.and(oneIsFalse, twoIsFalse, threeIsFalse, fourIsFalse)});

        params.add(new Object[]{Predicates.and(oneIsFalse, twoIsFalse, threeIsFalse, fourIsFalse), threeToTrue, Predicates.and(oneIsFalse, twoIsFalse, threeIsTrue, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsFalse, twoIsFalse, threeIsTrue, fourIsFalse), oneToTrue, Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsFalse), fourToTrue, Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsTrue)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsTrue), twoToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsTrue)});


        return params;
    }

    Predicate<SystemState> verifyStartState;
    Runnable changeState;
    Predicate<SystemState> verifyEndState;

    public SequentialParams(Predicate<SystemState> pre, Runnable task, Predicate<SystemState> post) {
      verifyStartState = pre;
      changeState = task;
      verifyEndState = post;
    }

    @Test
    public void perform() {
        Assert.assertTrue(verifyStartState.apply(state));
       changeState.run();
       Assert.assertTrue(verifyEndState.apply(state));
    }


    private static class SystemState {
        public boolean one = false;
        public boolean two = false;
        public boolean three = false;
        public boolean four = false;

    }

}
Jeremiah
  • 1,145
  • 6
  • 8