What is the role of asserts in C++ programs that have unit tests?

Question

I've been adding unit tests to some legacy C++ code, and I've run into many scenarios where an assert inside a function will get tripped during a unit test run. A common idiom that I've run across is functions that take pointer arguments and immediately assert if the argument is NULL.

I could easily get around this by disabling asserts when I'm unit testing. But I'm starting to wonder if unit tests are supposed to alleviate the need for runtime asserts. Is this a correct assessment? Are unit tests supposed to replace runtime asserts by happening sooner in the pipeline (ie: the error is caught in a failing test instead of when the program is running).

On the other hand, I don't like adding soft fails to code (e.g. if (param == NULL) return false;). A runtime assert at least makes it easier to debug a problem in case a unit test missed a bug.

Answer 1

A runtime assert at least makes it easier to debug a problem in case a unit test missed a bug.

This is a pretty fundamental point. Unit tests are not meant to replace assertions (which IMHO are a standard part of producing good quality code), they're meant to complement them.

Secondly, lets say you have a function Foo which asserts that it's parameters are valid.
In your unit test for Foo you can make sure you only supply valid parameters, so you think you're alright.
6 months down the track some other developer is going to call Foo from some new piece of code (which may or may not have unit tests), and at that point you'll be very grateful you put those asserts in there.

Answer 2

If your unit test code is correct, then the assert is a bug that the unit test has uncovered.

But it is far more likely that your unit test code is violating the constraints of the units it tests - your unit test code is buggy!

Commentators have raised the point that:

Consider a unit test that validates the function handles invalid input properly.

The assert is the programmer's way of handling invalid input, by aborting the program. By aborting, the program is functioning properly.

Asserts are only in the debug builds (they are not compiled if the NDEBUG macro is defined) and it's important to test that the program does something sensible in release builds. Consider running your invalid parameters unit-test on release builds.

If you want both worlds - checking asserts fire in debug builds - then you want your in-thread unit test harness to capture these asserts. You can do this by providing your own assert.h rather than using the system one; a macro (you'd want the __LINE__ and __FILE__ and ##expr) would call a function you wrote, which can throw a custom AssertionFailed when run in a unit test harness. This obviously does not capture asserts compiled into other binaries you link against but did not compile from source with your custom asserter. (I'd recommend this over providing your own abort(), but that's another approach you might consider to achieve the same end.)

Answer 3

Assertions catch when your code is being used incorrectly (violating its constraints or preconditions - using a library without initialization, passing NULL to a function that won't accept it, etc.).

Unit tests verify that your code does the right thing as long as it's being used correctly.

Assertions also catch when your code has entered a state which you believed to be impossible (which, since it is believed to be impossible, can't be unit tested).

Interestingly, at least one C++ unit testing framework (Google Test) supports death tests, which are unit tests that verify that your assertions work properly (so that you can know your assertions are doing their job of catching invalid program states).

Answer 4

IMO asserts and unit tests are fairly independent concepts. Neither of them can replace the other.

Asserts are meant to ensure that certain conditions / invariants are always valid during the lifetime of the program. Or to be more precise, to ensure that if such a condition gets broken, we get to know about it ASAP, as close to the root cause of the problem as possible.

Unit tests are meant to ensure that certain parts of the code work properly in isolation from the rest of the program.

You can't ensure by unit testing a class that its environment is always going to fulfill its part of the contract under real life circumstances. More so as said environment includes future developers who might not have a clue about the interface contract governing the usage of this class (be it implicit or carefully documented). And also a host of other software and hardware components, which may change and/or get broken anytime, in a manner uncontrollable by the developers of this specific program.

Answer 5

Usually, you should not be able to trip asserts as they are supposed to catch "impossible situations." If an assert fires, that should indicate a bug.

One exception to this rule is that many developers use asserts to verify that arguments are valid. This is okay provided there is also a backup for builds without asserts:

assert(arg != 0);
if (arg != 0)
    throw std::runtime_error();

This way, if a bad argument only happens under specific conditions (i.e. out in the field), it will still be caught.

If you code this way, you can turn off asserts and write negative tests to make sure bad arguments are caught.

Answer 6

Two possibilities here:

1) The behaviour of the function is defined (by its own interface explicitly, or by general rules for the project) when the input is null. The unit test therefore needs to test this behaviour. So you need a handler to run a process that runs the test case, and the handler validates that the code tripped the assertion and aborted, or you need to mock assert somehow.

2) The behaviour of the function is not defined when the input is null. The unit test therefore needs to not pass in null - a test is a client of the code too. You can't test something if there's nothing in particular that it's supposed to do.

There is no third option, "the function has undefined behaviour when passed a null input, but the tests pass in null anyway, just in case something interesting happens". So I don't see how, "I could easily get around this by disabling asserts when I'm unit testing" helps at all. Surely the unit tests will cause the function under test to dereference a null pointer, which isn't any better than tripping an assert. The whole reason the asserts are there is to stop something even worse from happening.

In your case, perhaps (1) applies in DEBUG builds, and (2) applies in NDEBUG builds. So perhaps you could run the null-input tests only on debug builds, and skip them when testing the release build.

Answer 7

I only use asserts to check for things that "can never happen". If an assert fires, then a programming mistake has been made somewhere.

Let's say a method takes the name of an input file, and a unit test feeds it the name of a non-existent file to see if a "file not found" exception is thrown. That's not something that "can never happen". It is possible to not find a file at runtime. I would not use an assert in the method to verify that the file was found.

However, the string length of the file name argument must never be negative. If it is, then there is a bug somewhere. So, I might use an assert to say "this length can never be negative". (It's just an artificial example.)

In the case of your question, if the function asserts != NULL, either the unit test is wrong and should not be sending in NULL, because this will never happen, or, the unit test is valid and NULL might possibly be sent in, and the function is wrong and should not be asserting != NULL and must instead handle that condition.

Answer 8

Personally I don't tend to use asserts as, as you've discovered, they often don't play nicely with unit tests. I tend to prefer throwing exceptions in situations where others would often use asserts. These checks, and the exceptions that are thrown on failure, are present in both debug and release builds and I find that they often catch things that 'can't possibly happen' even in release builds (which asserts often don't as they're often compiled out). I find it works better for me and means that I can write unit tests that expect the exception to be thrown on invalid input rather than expecting an assertion to fire.

Lots of people don't agree, see 1, 2, etc but I don't care. Avoiding asserts and using exceptions instead works well for me and helps me to produce robust code for clients...

Answer 9

1- Assert is a good way to clarify invariants (loop invariants) during development of algorithms. It can be useful for "readability" as well as debugging. The language Eiffel by Bertrand Meyer has a keyword invariant. In other languages, assert can be used for this purpose.

2- Assert can also be used in other circumstances as an intermediate solution during development, and removed gradually while the code is being completed. That is, as TODO items. Some of the asserts (those not in item #1 above) need to be replaced by exception handling, etc. It is much easier to spot them if all such checks are shown as asserts.

3- I sometimes use it to clarify the type of inputs in languages that don't have type checking system (Python and JavaScript), in certain contexts (e.g. when developing new algorithms). Not sure if it is a recommended practice. Like item #1, it is about increasing the readability of program.

Answer 10

First, for a unit test to hit an assert (or an ASSERT or _ASSERT or _ASSERTE on Windows builds), the unit test would need to run the code under test with the debug build.

I guess this can easily happen on a developer's machine. For our nightly builds, we only run the unit tests in the release configuration, so there's no worries about asserts there.

Second, one can take the normative approach with asserts --

Asserts are meant to ensure that certain conditions / invariants are always valid during the lifetime of the program. Or to be more precise, to ensure that if such a condition gets broken, we get to know about it ASAP, as close to the root cause of the problem as possible.

In this case, no unit test should raise an assertion, because calling the code in a way that an assertion is raised should not be possible.

or one can take the "pragmatic" approach with assertions:

Let developers sprinkle ASSERT all over the place for "don't do this" and "not implemented" scenarios. (And we can argue all day whether that's wrong™ or right™, but that won't get the features delivered.)

If you take the pragmatic approach, then a unit test hitting an assertion means the unit test called the code in a way that is not quite supported by the code. It just may mean that the code "does nothing" in a release build or it may mean that the code crashes in a release build or it may mean that the code does "something interesting".

Here's the options that I have been known to use:

• If the assert is accompanied by an additional check to make the call "harmless", make the unit test test for the assertion (in debug) and for the "harmless" condition in release.
• For crashes or "something interesting", either there's no unit test that makes sense, or you can make a "debug only" unit test that tests that you really get an assertion (though I'm not so sure that's helpful).