Open code review: Catch

by Max Galkin

This is one of the “open code review” posts, where I publish my notes after looking through and playing with one of the open source C++ projects. My main goal here is to become a better coder by learning from the experience of other developers, and my secondary goal is to build a mental map of the tools and frameworks available “out there” to not reinvent the proverbial wheel, should I ever need one. The blog post expresses my personal opinion, not affiliated, endorsed, sponsored, etc. I am not arguing for or against the usage of any specific open source library. I will be grateful if you take time to point out any misunderstanding I might have.

Today I looked at philsquared/Catch, “a multi-paradigm automated test framework for C++ and Objective-C (and, maybe, C)”.

Actually, it was long on my list of “to-read” projects, since the first time I’ve heard of it was at the “Meeting C++” conference in December ’14, where Phil Nash, its author, talked about it.

So, why bother about this “yet another” test framework? Catch dedicates a page to the answer, but here’s what caught my attention:

  • extreme simplicity and minimalism of the design (you need to learn very few things to start writing Catch tests);
  • plain text snippets as “first-class” citizens in test names and descriptions (so the tests look like a spec, and test assertions give richer description of the failed requirement);
  • an inventive way to get rid of setup code duplication via the mechanism of “sections” (saves you some coding, see below an example of how sections work).

Here’s how a trivial test in Catch looks like, from its tutorial:

  #define CATCH_CONFIG_MAIN  // This tells Catch to provide a main() - only do this in one cpp file
  #include "catch.hpp"

  unsigned int Factorial(unsigned int number) {
    return number <= 1 ? number : Factorial(number - 1)*number;
  }

  TEST_CASE("Factorials are computed", "[factorial]") {
    REQUIRE(Factorial(1) == 1);
    REQUIRE(Factorial(2) == 2);
    REQUIRE(Factorial(3) == 6);
    REQUIRE(Factorial(10) == 3628800);
  }

 

And here’s my somewhat unorthodox demo of Catch “sections”. I’m trying to illustrate the mechanism of the section tree traversal. This is not a test at all, I’m using Catch sections here to print 4 lines from Shakespeare which share some of their first few words starting with “Thou shalt…” :) For a more practical example see the vector tests in the Catch tutorial.

  #define CATCH_CONFIG_MAIN
  #include "catch.hpp"

  #include <iostream>
  #include <string>
  using namespace std;

  // Catch "sections" are "forks" in the test execution tree.
  // A test run traverses all possible paths to the leaves.
  // This test case produces 4 strings by traversing 4 paths. 
  TEST_CASE("4 lines from Shakespeare")
  {
    cout << "Thou shalt ";  // http://www.rhymezone.com/r/ss.cgi?q=thou+shalt
    
    SECTION("...not")
    {
      cout << "not ";
      
      SECTION("")
        cout << "stir a foot to seek a foe. (Romeo and Juliet: I, i)" << endl;
      
      SECTION("")
        cout << "sigh, nor hold thy stumps to heaven, (Titus Andronicus: III, ii)" << endl;
    }

    SECTION("")
      cout << "remain here, whether thou wilt or no. (A Midsummer Night's Dream: III, i)" << endl;
    
    SECTION("")
      cout << "continue two and forty hours, (Romeo and Juliet: IV, i)" << endl;
  }

  // Test output:

  //Thou shalt not stir a foot to seek a foe. (Romeo and Juliet: I, i)
  //Thou shalt not sigh, nor hold thy stumps to heaven, (Titus Andronicus: III, ii)
  //Thou shalt remain here, whether thou wilt or no. (A Midsummer Night's Dream: III, i)
  //Thou shalt continue two and forty hours, (Romeo and Juliet: IV, i)
  //===============================================================================
  //test cases: 1 | 1 passed
  //assertions: - none -


A few cons I’m aware of all stem from the fact that the framework is macro-based:

  • As Phil points out in this blog post, stepping through the macro-based assertions in debugger can be painful (but then maybe you should just put more assertions in the test to avoid using debugger at all?);
  • Some C++ refactoring tools tend to choke on code inside macros, so you might need to manually update the references, if you rename something (but C++ coders aren’t used to a working refactoring anyway :) ).

Now, what about the implementation? The implementation is quite neat, actually, and, as one would expect, Catch uses itself to unit test itself.

Let’s talk about the magic first, the REQUIRE(<expr>) macro splits the expression inside to show you what the sides evaluated to. Phil explains the mechanics behind this in his talk: basically the REQUIRE(<expr1> <op> <expr2>) line is rewritten as (<MagicClass> ->* <expr1> <op> <expr2>), and the ->* operation takes precedence over all the comparison operations and thus the MagicClass first “eats” the <expr1>, and then the <expr2> using the appropriate overloaded operator. In some cases, when <expr1> is too complicated, the precedence can be messed up, but all you need to do is to take <expr1> in parentheses to fix this (or separate it into several statements and assertions).

I also like how everything in code is readable, because it is named very clearly and makes a good use of enums (scoped by structs, because it is C++98 by default).

  // ...
    bool includeSuccessfulResults() const { return m_data.showSuccessfulTests; }
    bool warnAboutMissingAssertions() const { return m_data.warnings & WarnAbout::NoAssertions; }
    ShowDurations::OrNot showDurations() const { return m_data.showDurations; }
    RunTests::InWhatOrder runOrder() const  { return m_data.runOrder; }
  // ...

  struct OnUnusedOptions { enum DoWhat { Ignore, Fail }; };

  int applyCommandLine(
    int argc, 
    char* const argv[],
    OnUnusedOptions::DoWhat unusedOptionBehaviour = OnUnusedOptions::Fail )
  {
    // ...
  }       

 

Overall, I think this framework might be ideal for new projects as it is very lightweight and easy to use, but it can also be integrated into existing code bases, because it supports major unit-testing framework output formats as well as extendable custom formatting.