335

I know there are already a few questions regarding recommendations for C++ unit test frameworks, but all the answers did not help as they just recommend one of the frameworks but do not provide any information about a (feature) comparison.

I think the most interesting frameworks are CppUnit, Boost and the new Google testing framework. Has anybody done any comparison yet?

Cœur
  • 37,241
  • 25
  • 195
  • 267
housemaister
  • 3,918
  • 3
  • 19
  • 13
  • I have my own IOC based testing framework which I like more because it isn't just a clone of what all the others do but addresses what I find all the problems of the others to be. You write test cases by deriving from a class, not by using macros. Macros only used for assertions as they give you reflection. Customised output of testing statistics. Run from IOC scripting so you choose what you test, how often and with what parameters. – CashCow Oct 23 '14 at 15:48
  • and it's brilliant from a development point of view as when I add my own test I can run it without having to run everyone else's at the same time. So I know that my code is working. – CashCow Oct 23 '14 at 15:49

9 Answers9

132

A new player is Google Test (also known as Google C++ Testing Framework) which is pretty nice though.

#include <gtest/gtest.h>

TEST(MyTestSuitName, MyTestCaseName) {
    int actual = 1;
    EXPECT_GT(actual, 0);
    EXPECT_EQ(1, actual) << "Should be equal to one";
}

Main features:

  • Portable
  • Fatal and non-fatal assertions
  • Easy assertions informative messages: ASSERT_EQ(5, Foo(i)) << " where i = " << i;
  • Google Test automatically detects your tests and doesn't require you to enumerate them in order to run them
  • Make it easy to extend your assertion vocabulary
  • Death tests (see advanced guide)
  • SCOPED_TRACE for subroutine loops
  • You can decide which tests to run
  • XML test report generation
  • Fixtures / Mock / Templates...
Wernight
  • 36,122
  • 25
  • 118
  • 131
  • 3
    I really enjoy using google test over some of the other frameworks especially with its mocking capabilities that can be found in the googlemock framework. – Mike Oct 26 '10 at 05:27
  • 8
    I provide all these features (although some are not yet public) and more in my new test framework, CATCH. See my answer for link. – philsquared Dec 28 '10 at 03:14
  • 2
    combining it together with Google C++ Mocking framework makes it really powerful xUnit test framework for unit test C++ code. – ratkok Jun 07 '11 at 18:38
  • 5
    @CashCow Running with the build is something different from test detection. Running with build depends on your build system. Test detection means you don't **have** to list all tests in another class, just create the tests methods and that's it. – Wernight Nov 03 '14 at 14:51
  • 1
    I dislike their overuse of macros though, and the fact that use common words like TEST which might clash with something. GTEST would be better, less likely to clash. – CashCow Nov 05 '14 at 17:24
  • @CashCow I would keep tastes outside SO (and you can rename if you really wanted to). Macros are ugly I admit but probably a necessary evil in C++ to support all the features without scaffolding code. – Wernight Nov 07 '14 at 12:54
  • By Scaffolding code you mean deriving from their classes by actually typing their names? What's wrong with that. That's how you normally use an API and it is a clear indication of what you are doing. Macros are useful for reflection in the asserts and to add the __FILE__ and __LINE__ for you. It is however totally non-intuitive to know what TEST and TEST_F mean and these things don't work if you do them wrong. For example, can I use their asserts in free-functions called from my test functions? – CashCow Nov 10 '14 at 16:12
  • @CashCow Remember C++ doesn't have the kind of reflection most language have. I suggest you reach me directly by email/IM so we can discuss more in details if you like. – Wernight Nov 11 '14 at 08:31
  • I know C++ doesn't have reflection and therefore macros are useful for the asserts. I have written a testing library. It uses dependency injection like GTEST does. It has macros for asserts but not for anything else, and there's a Predicate class in the API. The macros are there for convenience. You have to #include their header to use them, they are not included for you. – CashCow Nov 11 '14 at 10:02
  • I second this. Currently using Boost.Test and suffering from lack of more assert operations and a way to properly output xUnit conforming xml files to nicely integrate with TeamCity. – Samuel Aug 11 '16 at 08:15
129

I've just pushed my own framework, Catch2, out there. It's still under development but I believe it already surpasses most other frameworks. Different people have different criteria but I've tried to cover most ground without too many trade-offs. Take a look at my linked blog entry for a taster. My top five features are:

  • Header only
  • Auto registration of function and method based tests
  • Decomposes standard C++ expressions into LHS and RHS (so you don't need a whole family of assert macros).
  • Support for nested sections within a function based fixture
  • Name tests using natural language - function/method names are generated

It also has Objective-C bindings. The project is hosted on Github

VLL
  • 9,634
  • 1
  • 29
  • 54
philsquared
  • 22,403
  • 12
  • 69
  • 98
  • 3
    [doctest](https://github.com/onqtam/doctest) is my reimplementation of Catch with a huge focus on compilation speed - checkout the [FAQ](https://github.com/onqtam/doctest/blob/master/doc/markdown/faq.md#how-is-doctest-different-from-catch) to see how they differ – onqtam Oct 10 '16 at 19:34
  • 3
    I'm really at a loss when comparing all the testing frameworks (one of which I now have to choose). Would you write your own answer comparing and contrasting doctest with Catch and other offerings? – einpoklum Nov 04 '16 at 11:48
  • @einpoklum I would if I could - but the question is closed as being ```not constructive``` and I can write only a comment... – onqtam Apr 23 '17 at 07:11
  • You should add: no fall through on failures, meaning the tests will stop immediately. Google test proudly declare to be exceptions free, but they fall through on failures if you don't check return codes (at least this is the default behavior), meaning that the test continues if you don't do it. I found this laughable. – ceztko Oct 30 '21 at 14:02
108

See this question for some discussion.

They recommend the articles: Exploring the C++ Unit Testing Framework Jungle, By Noel Llopis. And the more recent: C++ Test Unit Frameworks

I have not found an article that compares googletest to the other frameworks yet.

Community
  • 1
  • 1
Sam Saffron
  • 128,308
  • 78
  • 326
  • 506
  • 1
    As I wrote: all the answers just recomend one of the frameworks but do not compare the framework to another. – housemaister Oct 28 '08 at 11:21
  • You're not happy with the article either ? – Gishu Oct 28 '08 at 11:33
  • 8
    One criticism: the article, while good, is from 2004 and doesn't include Google Test. – richq Oct 28 '08 at 11:54
  • 2
    In the first link you'll see two comparisons. Except the new framework from google, most information is (are?) still relevant. (And CppUnit is not the most interesting, it's too clumsy to use) – Luc Hermitte Oct 28 '08 at 12:55
  • 1
    fixed up links and expanded the answer with a more recent comparison – Sam Saffron Oct 28 '08 at 21:13
  • The second link is broken. Could you please fix it? – Yuchen May 21 '14 at 21:53
  • Answers shouldn't include just a link to see something. it's now common in the community to include full info, since all links will become invalid at sometime throughout the great eternal test of time, where at some time all things end, and all things begin, including us and our descendants. – PathToLife Jun 16 '20 at 08:04
  • The link to the more recent link resource is outdated. It states that boost does not allow to select tests from the command line which is no longer true. Not only can you select tests to run from the command line, but you can also do so using patterns. – fabian Jul 13 '21 at 19:09
55

Boost Test Library is a very good choice especially if you're already using Boost.

// TODO: Include your class to test here.
#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>

BOOST_AUTO_TEST_CASE(MyTestCase)
{
    // To simplify this example test, let's suppose we'll test 'float'.
    // Some test are stupid, but all should pass.
    float x = 9.5f;

    BOOST_CHECK(x != 0.0f);
    BOOST_CHECK_EQUAL((int)x, 9);
    BOOST_CHECK_CLOSE(x, 9.5f, 0.0001f); // Checks differ no more then 0.0001%
}

It supports:

  • Automatic or manual tests registration
  • Many assertions
  • Automatic comparison of collections
  • Various output formats (including XML)
  • Fixtures / Templates...

PS: I wrote an article about it that may help you getting started: C++ Unit Testing Framework: A Boost Test Tutorial

Wernight
  • 36,122
  • 25
  • 118
  • 131
  • I used to use Boost test and liked it except that it seemed to change significantly between release. It was difficult enough selling unit testing to my client without having to spend more of my time (and their money) fixing the tests when the API changed, than fixing the code it was meant to be testing. In the end I ditched it and wrote my own - this was about 5 years ago though. – Component 10 Oct 10 '12 at 08:51
27

Wikipedia has a comprehensive list of unit testing frameworks, with tables that identify features supported or not.

John Deters
  • 4,295
  • 25
  • 41
16

I've recently released xUnit++, specifically as an alternative to Google Test and the Boost Test Library (view the comparisons). If you're familiar with xUnit.Net, you're ready for xUnit++.

#include "xUnit++/xUnit++.h"

FACT("Foo and Blah should always return the same value")
{
    Check.Equal("0", Foo()) << "Calling Foo() with no parameters should always return \"0\".";
    Assert.Equal(Foo(), Blah());
}

THEORY("Foo should return the same value it was given, converted to string", (int input, std::string expected),
    std::make_tuple(0, "0"),
    std::make_tuple(1, "1"),
    std::make_tuple(2, "2"))
{
    Assert.Equal(expected, Foo(input));
}

Main features:

  • Incredibly fast: tests run concurrently.
  • Portable
  • Automatic test registration
  • Many assertion types (Boost has nothing on xUnit++)
  • Compares collections natively.
  • Assertions come in three levels:
    • fatal errors
    • non-fatal errors
    • warnings
  • Easy assert logging: Assert.Equal(-1, foo(i)) << "Failed with i = " << i;
  • Test logging: Log.Debug << "Starting test"; Log.Warn << "Here's a warning";
  • Fixtures
  • Data-driven tests (Theories)
  • Select which tests to run based on:
    • Attribute matching
    • Name substring matchin
    • Test Suites
moswald
  • 11,491
  • 7
  • 52
  • 78
  • 4
    The question is asking for comparison. IMO, it is vital to present what are the *differences* between your framework and, at least, the two popular ones: googletest and Boost. Especially, if you advertise xUnit++ as alternative to those two. Will be +1 if updated :) – mloskot Feb 04 '13 at 18:00
  • 1
    Fair enough. :) I've already got a [comparison table](https://bitbucket.org/moswald/xunit/wiki/Compare.wiki) on the [wiki](https://bitbucket.org/moswald/xunit/wiki/Home), but I will try to sum up a few of the differences directly in my answer. – moswald Feb 04 '13 at 20:11
  • 1
    I decided to just link the wiki table directly, it was cluttering up the summary to list it all out. – moswald Feb 04 '13 at 20:24
  • Also written in macro language.. What's wrong with using C++ instead of FACT and THEORY macros? – CashCow Nov 05 '14 at 17:26
  • Personally, I really hate macros. I've considered writing v2.0 using method calls and lambdas, I just don't write any C++ these days and haven't gotten around to it. – moswald Nov 05 '14 at 18:17
  • 1
    has your project been discontinued? Last commit dates back to 09/2015... Anyway, great answer. Thanks. – zertyz Oct 03 '19 at 12:41
5

CppUTest - very nice, light weight framework with mock libraries. Worthwhile taking a closer look.

ratkok
  • 739
  • 9
  • 15
4

CPUnit (http://cpunit.sourceforge.net) is a framework that is similar to Google Test, but which relies on less macos (asserts are functions), and where the macros are prefixed to avoid the usual macro pitfall. Tests look like:

#include <cpunit>

namespace MyAssetTest {
    using namespace cpunit;

    CPUNIT_FUNC(MyAssetTest, test_stuff) {
        int some_value = 42;
        assert_equals("Wrong value!", 666, some_value);
    }

    // Fixtures go as follows:
    CPUNIT_SET_UP(MyAssetTest) {
        // Setting up suite here...
        // And the same goes for tear-down.
    }

}

They auto-register, so you need not more than this. Then it is just compile and run. I find using this framework very much like using JUnit, for those who have had to spend some time programming Java. Very nice!

Community
  • 1
  • 1
Roger
  • 41
  • 1
2

API Sanity Checker — test framework for C/C++ libraries:

An automatic generator of basic unit tests for a shared C/C++ library. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files.

The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging.

Unique features in comparison with CppUnit, Boost and Google Test:

  • Automatic generation of test data and input arguments (even for complex data types)
  • Modern and highly reusable specialized types instead of fixtures and templates
Community
  • 1
  • 1
linuxbuild
  • 15,843
  • 6
  • 60
  • 87