Assert multiple conditions in a single test, or split into multiple tests?

Question

If you were testing a count function like the one below, is it considered to be 'right' or 'wrong' to test multiple things for the function in one function vs having a test function for each of the tests?

function testGetKeywordCount() {
    $tester = $this -> getDatabaseTester($this -> CompleteDataFile);
    $tester -> onSetUp();

    $KeywordID = 0;
    $this -> setExpectedException('InvalidArgumentException');
    $this -> keyword -> getKeywordCount($KeywordID,'Active');

    $KeywordID = 1;
    $this -> setExpectedException('InvalidArgumentException');
    $this -> keyword -> getKeywordCount($KeywordID,'InvalidStatus');

    $this -> assertEquals(1, $this -> keyword -> getKeywordCount($KeywordID,'Active'));

    $tester -> onTearDown();
}

Answer 1

You should have multiple test functions, where each tests its own condition. That way it is easier to spot a failure without debugging.

Answer 2

Having one test case for each scenario is ideal. However, in some cases it is more convenient (efficient from implementation effort point of view) to test more than one scenario in one test case. If you use a framework that doesn't stop on first failure, but tries to execute as much as possible in a test case, that framework is appropriate for multiple scenarios per test case.

I prefer to spend as less time as possible on unit testing, and still get as most coverage as possible in that time.

In the end, it matters less how you implement the unit test, but more the correctness of those tests.

Answer 3

One argument for splitting the assertions into two separate tests is that, if one of the assertions fails, you'll get one failure; if both assertions fail, you'll get two failures.

Also, by making the name of each test as suggestive as possible, you'll get extra clues when something breaks.

Answer 4

Testing frameworks don't always make it worth your effort to follow the one assertion per test rule.

One that does is RSpec for Ruby, which allows you to set up nested example groups. For example:

• A User
  • without a password
    • is invalid
    • throws exception
  • with a password
    • that has been used before
      • is invalid
      • throws exception
      • trips security warning
    • that hasn't been used before
      • is valid
      • redirects to account page

By progressively building up scenarios and testing each step along the way, its easier to stick to the one assertion per test approach. It also makes it easier to spot untested scenarios.

Answer 5

This is a question with different answers depending on who you ask it and depends mainly on his/her personal opinion or professional experience. Some ultra theoretical people will tell you that having more than one assert in a test is a supreme sin and you will be doomed forever. But others with more real world experience may tell you that there are situations when it is perfectly good to have 10 or even 50 asserts per test.

So, who is right?

I always try to be objective when facing this kind of dilemas, and to take a choice I decided to perform a small research with some of the most popular github repos that are being developed today by certified and experienced professionals.

So, how do big players test their own projects? And more important, how are the unit testing frameworks being unit tested themselves?

Let's see a few examples:

Hibernate

https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/test/java/org/hibernate/engine/spi/EntityEntryTest.java https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/test/java/org/hibernate/engine/spi/NonSortedExecutableListTest.java https://github.com/hibernate/hibernate-orm/blob/master/hibernate-core/src/test/java/org/hibernate/engine/spi/SortedExecutableListTest.java https://github.com/hibernate/hibernate-orm/blob/master/hibernate-envers/src/test/java/org/hibernate/envers/test/JpaStaticMetamodelTest.java https://github.com/hibernate/hibernate-orm/blob/master/hibernate-hikaricp/src/test/java/org/hibernate/test/hikaricp/HikariCPConnectionProviderTest.java https://github.com/hibernate/hibernate-orm/blob/master/hibernate-proxool/src/test/java/org/hibernate/test/proxool/ProxoolConnectionProviderTest.java

Spring

https://github.com/spring-projects/spring-framework/blob/master/spring-core/src/test/java/org/springframework/util/AntPathMatcherTests.java https://github.com/spring-projects/spring-framework/blob/master/spring-core/src/test/java/org/springframework/util/ClassUtilsTests.java https://github.com/spring-projects/spring-framework/blob/master/spring-core/src/test/java/org/springframework/util/ObjectUtilsTests.java https://github.com/spring-projects/spring-framework/blob/master/spring-core/src/test/java/org/springframework/util/AutoPopulatingListTests.java

junit 4

https://github.com/junit-team/junit4/blob/master/src/test/java/junit/tests/extensions/ActiveTestTest.java https://github.com/junit-team/junit4/blob/master/src/test/java/junit/tests/extensions/RepeatedTestTest.java https://github.com/junit-team/junit4/blob/master/src/test/java/junit/tests/runner/ResultTest.java https://github.com/junit-team/junit4/blob/master/src/test/java/org/junit/rules/TimeoutRuleTest.java

junit 5

https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/java/org/junit/platform/launcher/core/DefaultLauncherTests.java https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/java/org/junit/platform/launcher/core/LauncherDiscoveryRequestBuilderTests.java https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/java/org/junit/platform/launcher/listener/SummaryGenerationTests.java https://github.com/junit-team/junit5/blob/master/junit-jupiter-engine/src/test/java/org/junit/jupiter/engine/StandardTestClassTests.java https://github.com/junit-team/junit5/blob/master/junit-jupiter-engine/src/test/java/org/junit/jupiter/engine/DiscoveryFilterApplierTests.java

Google Truth

https://github.com/google/truth/blob/master/core/src/test/java/com/google/common/truth/DoubleSubjectTest.java https://github.com/google/truth/blob/master/core/src/test/java/com/google/common/truth/BigDecimalSubjectTest.java https://github.com/google/truth/blob/master/core/src/test/java/com/google/common/truth/DoubleSubjectTest.java

As we can see in the above examples, professional developers doesn't seem to care much about the single assert commandment. In fact, they break this rule most of the time and it looks like they are perfectly fine with it. Maybe they ignore it because is not a strict rule but just a recommendation. It is worth mentioning that even unit testing frameworks test themselves with more than one assert per test most of the time.

So my conclusion is clear: Regarding this concern, it is perfectly valid to have as many asserts as you want in a single test. If professional developers are doing it, you can do it too.

Answer 6

I would not talk about Unit tests in your example code above.
Your example is more an automated functional test, that tests a flow of functions.

BUT In this case it is fine to have multiple assertions.

Just make sure that 2 assertions are not one behind the other.

Bad example

public void ValidateRulesEntry_Valid_ValidConditionsFromFile()
{
    string condition = "Target.HasValue";
    string returnMessage;

    bool successFul = CodeParserTryParseCondition(condition, out returnMessage);


    Assert.IsTrue(successFul);
    Assert.IsFalse(string.IsNullOrEmpty(returnMessage));
    Assert.IsTrue(returnMessage == "OK");

}  

The 2 last assertions are dependent of the 1st assertion IsTrue(successFul).

Think about: If that test fails --> Tell me why (without looking into Debug output)