Why are my @BeforeTest methods sometimes reported as the last element of previous test?

Question

I have a testNG suite, executing some selenium code. I am testing navigation between pages in a webapp I have. There are four different user levels, which gives access to restricted pages, based on user type. I also need to test this in both Chrome and IE.

I have followed a Mkyong.com tutorial for my testng.xml setup.

I have chosen to collect all navigation tests for all users, and both browsers, in a single testng.xml. It gives a total of 8 test classes, where 4 and 4 have the same configuration class executed before tests are run.

testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Releaseboard Local tests" parallel="none">


    <test name="user1 chrome">
        <classes>
            <class name="testsuite.ChromeConfig" />
            <class name="testcase.User1NavigationTests" />
        </classes>
    </test>
    <test name="user2 chrome">
        <classes>
            <class name="testsuite.ChromeConfig" />
            <class name="testcase.User2NavigationTests" />
        </classes>
    </test>
    <test name="user3 chrome">
        <classes>
            <class name="testsuite.ChromeConfig" />
            <class name="testcase.User3NavigationTests" />
        </classes>
    </test>
    <test name="user4 chrome">
        <classes>
            <class name="testsuite.ChromeConfig" />
            <class name="testcase.User4NavigationTests" />
        </classes>
    </test>


    <test name="user1 ie">
        <classes>
            <class name="testsuite.IeConfig" />
            <class name="testcase.User1NavigationTests" />
        </classes>
    </test>
    <test name="user2 ie">
        <classes>
            <class name="testsuite.IeConfig" />
            <class name="testcase.User2NavigationTests" />
        </classes>
    </test>
    <test name="user3 ie">
        <classes>
            <class name="testsuite.IeConfig" />
            <class name="testcase.User3NavigationTests" />
        </classes>
    </test>
    <test name="user4 ie">
        <classes>
            <class name="testsuite.IeConfig" />
            <class name="testcase.User4NavigationTests" />
        </classes>
    </test>
</suite>

Now, when I run these tests, I expect this output for all 8 test classes:

It does not matter if they are red or green, but I expect all test tags from the xml to start two levels up, and always at the same level. There is also a level above the three shown, but it does not matter at this point.

The output I get however, is that sometimes the next level 2, shows up inside the previous level 1. It then fails the @BeforeTest annotated method. That would not be so bad in itself, because it actually also runs the test at level 1 after failing inside the previous test. However, if a test has shown up inside the previous, it always fails @BeforeTest at level 1 as well, and all level 3 test methods are automatically skipped. See image below:

All of the lines with time at the end are actual lines from the screenshot. The rest has been edited for anonymity.

Note how my @BeforeTest, and @AfterTest methods, setup and tearDown, are now listed as tests for some reason. Both inside the wrong test, without any real test methods, and inside the next, suddenly with the actual test methods. These errors appear completely random every time. There is no specific test that always contains the next one. Sometimes there are three of them, sometimes only one. The only thing consistent is that it is never two in a row. I really really do not understand what is going on here. I'd love to explain better, or do more research, but I'm having a really hard time boiling this problem down to a searchable term. I have a feeling this might go unanswered until i can put bounty on it.

I will post any other code examples anyone can think of wanting to see, but for now I would not know where to start.

Answer 1

It seems like the answer to my question, is that this is what happens when a @BeforeTest annotated method fails. In other words, the fact that the setup and teardown shows up inside the previous tests hierarchical structure, is likely just minor reporting error on testNG's side. It is not the cause of any error. The cause of the error, and the skipped tests, are in my case the result of an unstable setup method. The fact that it worked sometimes, and sometimes not, was due to sensitive small conditional elements on my pages, which were some times found/loaded in time, and some times not.