[toc]
One of the first decisions in a new project is which unit testing framework to use. Traditionally I’ve used CppUnit, so I pulled down the current release and started working.

This left me unhappy as the first test produced this compile-time error:

/usr/local/gcc-20140104/include/cppunit/TestAssert.h:109:6: note:   template argument deduction/substitution failed:
cpput_eval.cc:13:5: note:   deduced conflicting types for parameter ‘const T’ (‘int’ and ‘std::basic_string::size_type {aka long unsigned int}’)
     CPPUNIT_ASSERT_EQUAL(4, str.size());

For a couple days I worked around this by casting the integer literal to a type that satisfied the calls, but eventually I got fed up.

So I looked for alternatives. I found fault with the first two choices, but joy with the third. Herein are some examples with discussion of what they reveal about the choices. The files are available as a github gist.

The Test Criteria

Three specific assertions were found to cause trouble with various solutions, so the examples used below show all of them:

• Comparing a std::string size() with an integer literal;
• Pointer-equality testing for char * values;
• Comparing a floating point result to a specific absolute accuracy

In addition, these criteria are relevant:

• Verbosity: how much boilerplate do you have to add that isn’t really part of your test?
• Installation overhead: is it easy to build the library for specific compiler flags or is the assumption that you build it once and share it? This matters when playing with advanced language feature flags such as -std=c++1y, which can affect linking test cases together.
• Assertion levels: when a test fails can you control whether the test keeps going or aborts (e.g., when following assertions would be invalid if the first fails).
• Assertion comparisons: can you express specific relations (not equal, greater than) or is it mostly a true/false capability?

CppUnit

Originally on SourceForge, this project has developed new life at freedesktop.org.

CppUnit comes with a standard configure/make/make install build process which installs the headers and the support library into the proper directories within a toolchain prefix. You need to provide a main routine to invoke the test driver.

CppUnit provides only one level of assertion: the test case aborts when it fails. It also has limited ability to express specific requirements (for example, there is CPPUNIT_ASSERT_EQUAL(x,y) but no CPPUNIT_ASSERT_NOT_EQUAL(x,y).

Here’s what the tests looks like with CppUnit:

#include <cppunit/extensions/HelperMacros.h>
#include <string>
#include <cmath>

class testStringStuff : public CppUnit::TestFixture
{
protected:
  void testBasic ()
  {
    const char * const cstr{"no\0no\0"};
    const std::string str("text");
    CPPUNIT_ASSERT_EQUAL(std::size_t{4}, str.size());
    CPPUNIT_ASSERT(cstr != (cstr+3));
  }

private:
  CPPUNIT_TEST_SUITE(testStringStuff);
  CPPUNIT_TEST(testBasic);
  CPPUNIT_TEST_SUITE_END();
};

CPPUNIT_TEST_SUITE_REGISTRATION(testStringStuff);

class testFloatStuff : public CppUnit::TestFixture
{
protected:
  void testBasic ()
  {
    CPPUNIT_ASSERT_DOUBLES_EQUAL(11.045, std::sqrt(122.0), 0.001);
  }

private:
  CPPUNIT_TEST_SUITE(testFloatStuff);
  CPPUNIT_TEST(testBasic);
  CPPUNIT_TEST_SUITE_END();
};

CPPUNIT_TEST_SUITE_REGISTRATION(testFloatStuff);

There’s a lot of overhead, what with the need to define and register the suites, though it didn’t really bother me until I saw what other frameworks require. And I did have to do that irritating explicit cast to get the size comparison to compile.

The output is terse and all tests pass:

testFloatStuff::testBasic : OK
testStringStuff::testBasic : OK
OK (2)

Boost.Test

Boost is a federated collection of highly-coupled but independently maintained C++ libraries covering a wide range of capabilities. It includes Boost.Test, the unit test framework used by boost developers themselves.

Boost.Test can be used as a header-only solution, but I happened to install it in library form. This gave me a default main routine for invocation, though I did have to have a separate object file with preprocessor defines which incorporated it into the executable.

Boost.Test also supports three levels of assertion. WARN is a diagnostic only; CHECK marks the test as failing but continues; and REQUIRE marks the test as failing and stops the test. There are also a wide variety of conditions (EQUAL, NE, GT, …), each of which is supported for each level.

Here’s what the tests look like with Boost.Test:

#include <boost/test/unit_test.hpp>
#include <string>
#include <cmath>

BOOST_AUTO_TEST_CASE(StringStuffBasic)
{
  const std::string str("text");
  float fa[2];
  const char * const cstr{"no\0no\0"};
  BOOST_CHECK_EQUAL(4, str.size());
  BOOST_CHECK_NE(fa, fa+1);
  BOOST_CHECK_NE(cstr, cstr+3);
}

BOOST_AUTO_TEST_CASE(FloatStuffBasic)
{
  BOOST_CHECK_CLOSE(11.045, std::sqrt(122), 0.001);
}

This is much more terse than CppUnit, and seems promising. Here’s what happens when it runs:

Running 2 test cases...
butf_eval.cc(10): error in "StringStuffBasic": check cstr != cstr+3 failed [no == no]
butf_eval.cc(15): error in "FloatStuffBasic": difference{0.0032685%} between 11.045{11.045} and std::sqrt(122){11.045361017187261} exceeds 0.001%

*** 2 failures detected in test suite "Master Test Suite"

Um. Whoops?

Boost.Test silently treats the char* pointers as though they were strings, and does a string comparison instead of a pointer comparison. Which is not what I asked for, and not what BOOST_CHECK_NE(x,y) will do with other pointer types.

Boost.Test also does not provide a mechanism for absolute difference in floating point comparison. Instead, it provides two relative solutions: BOOST_CHECK_CLOSE(v1,v2,pct) checks that v1 and v2 are no more than pct percent different (e.g. 10 would be 10% different), while BOOST_CHECK_CLOSE_FRACTION(v1,v2,frac) does the same thing but using fractions of a unit (e.g. 0.1 would be 10% different). Now, you can argue that there’s value in a relative error calculation. But to have two of them, and not have an absolute error check—that doesn’t work for me.

Boost.Test also has a few other issues. The released version has not been updated for four years, but the development version used internal to the Boost project has many changes, which are expected to be released at some point in the future. From comments on the boost developers mailing list the documentation is generally agreed to be difficult to use, and has produced a rewritten version (which, honestly, is what I had to use to try it out).

All in all, I don’t feel comfortable depending on Boost.Test.

Google Test

Google Test is another cross-platform unit test framework, which supports a companion mocking framework to support unit testing of capabilities that are not stand-alone.

The code comes with configure/make/install support, but also provides a single-file interface allowing it to be built easily within the project being tested with the same compiler and options as the code being tested. You do need a separate main routine, but it’s a two-liner to initialize the tests and run them all.

Google Test supports two levels of assertion: failure of an ASSERT aborts the test, while failure of EXPECT fails the test but continues to check additional conditions. It also provides a wide variety of conditions.

Here’s what the tests look like with Google Test:

#include <gtest/gtest.h>
#include <string>
#include <cmath>

TEST(StringStuff, Basic)
{
  const std::string str("text");
  const char * const cstr{"no\0no\0"};
  ASSERT_EQ(4, str.size());
  ASSERT_NE(cstr, cstr+3);
}

TEST(FloatStuff, Basic)
{
  ASSERT_NEAR(11.045, std::sqrt(122.0), 0.001);
}

Even more terse than Boost.Test, because it doesn’t use something like GTEST_TEST or GTEST_ASSERT_EQ. To avoid conflict with user code I normally expect framework tools to provide their interfaces within a namespace (literally for C++, or by using a standard identifier prefix where that wouldn’t work). Both CppUnit and Boost.Test do this for their macros, but for unit test code that doesn’t get incorporated into an application I think it’s ok that this isn’t done.

And here’s what you get when running it:

[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from StringStuff
[ RUN      ] StringStuff.Basic
[       OK ] StringStuff.Basic (0 ms)
[----------] 1 test from StringStuff (0 ms total)

[----------] 1 test from FloatStuff
[ RUN      ] FloatStuff.Basic
[       OK ] FloatStuff.Basic (0 ms)
[----------] 1 test from FloatStuff (0 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 2 tests.

A little more verbose than I’m accustomed to from CppUnit, but it’s tolerable. The most important bit is the last line tells you the overall success, so you only need to scroll up if something didn’t work.

Conclusions

Summarizing the individual tests for each criterion, with a bold answer being preferable from my subjective perspective:

Feature	CppUnit	Boost.Test	Google Test
Handles size_t/int compares	no	yes	yes
Handles char* compares	yes	no	yes
Handles absolute float delta	yes	no	yes
Verbosity	high	low	low
Installation	toolchain	header-only or toolchain	project
Assertion Levels	one	three	two
Assertion Conditions	few	every	many

So I’m now happily using Google Test as the unit test framework for new C++ projects.

In fact, I’ve also started to use Google Mock, which turns out to be even more cool and eliminates the biggest limitation on unit testing: what to do if the routine being tested normally needs a heavy-weight and uncontrollable supporting infrastructure to satisfy its API needs. But I can’t really add anything beyond what you’ll can find on their wiki, so will leave it at that.