Increase your QTest productivity

The QTestLib framework is a tool for unit testing Qt based applications and libraries. I find it precious and simple to use, though it lacks some important features, supported, for example, by GTest. I’m not talking about mocking – for this you generally need an out-and-out framework (like GMock) – instead, I’m referring to simple things like fast deploying. Suppose you have written a test class like this:

#ifndef WIDGETTEST_H
#define WIDGETTEST_H

#include <QObject>

class WidgetTest : public QObject
{
 Q_OBJECT
private slots:
 void My_first_test();
 void On_button_pressed_should_save_the_world();
};

#endif //WIDGETTEST_H

The simplest approach to run all tests is to call QTest::qExec in your main:

#include "WidgetTest.h"
int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    WidgetTest wtest;
    return QTest::qExec(&wtest, argc, argv);
}

Now what if you add a new test class? This approach brings you about:

• Adding a new test (ok, this is normal),
• Including its header in main.cpp (bad),
• Laying a hand on the code in main.cpp (super bad), maybe xor-ing all the results.

This can be a pain for your productivity – before, the term “deploying” was related to “up and running on continuous integration systems” (e.g. TeamCity).

So, the first thing is about this. How to provide a simple and fast way to add new test classes with minimal impact? I’d like to write support code once.

The approach I currently use is this:

• I have a test runner that holds a list with all the test objects to run,
• each test class includes this runner and uses a special macro to add itself to the list,
• main.cpp includes the runner and uses another macro to run all tests.

Let me show you the code:

#ifndef TESTRUNNER_H
#define TESTRUNNER_H

// Qt includes
#include <QTest>
#include <QSharedPointer>
// std includes
#include <algorithm>
#include <list>
#include <iostream>

//////////////////////////////////////////////////////////////////////////
// Test Runner allows automatic execution of tests
class TestRunner
{
public:

static TestRunner& Instance()
{
   static TestRunner instance;
   return instance;
}

template <typename T>
char RegisterTest(char* name)
{
   if ( std::find_if( begin(m_tests), end(m_tests), [&name](QSharedPointer<QObject>& elem)
   { return elem->objectName() == name; }) == end(m_tests) )
    {
      QSharedPointer<QObject> test(new T());
      test->setObjectName(name);
      m_tests.push_back(test);
   }
   return char(1);
}

int RunAll(int argc, char *argv[])
{
   int errorCode = 0;
   std::for_each( begin(m_tests), end(m_tests), [&] (QSharedPointer<QObject>& test)
   {
      errorCode |= QTest::qExec(test.data(), argc, argv);
      std::cout << std::endl;
   } );

   return errorCode;
}

private:
   std::list<QSharedPointer<QObject>> m_tests;
};

// Use this macro after your test declaration
#define DECLARE_TEST(className)\
    static char test_##className = TestRunner::Instance().RegisterTest<className>(#className);

// Use this macro to execute all tests
#define RUN_ALL_TESTS(argc, argv)\
    TestRunner::Instance().RunAll(argc, argv);

#endif // TESTRUNNER_H

The trick is to define static variables (chars in this case) that will be constructed before the program starts (registering the tests to the runner). Instead of chars (that I know, it seems like a hack, it’s not a good design), we could use another object that handles the registration during its construction:

template <class T>
class TestAdder
{
public:
 TestAdder(const QString& name)
 {
   auto newTest = new T();
   newTest->setObjectName(name);
   TestRunner::Instance().RegisterTest(newTest);
 }
};

TestRunner defines a simpler function:

void RegisterTest(QObject* test)
{
   auto testName = test->objectName();

   if ( std::find_if(begin(m_tests), end(m_tests), [&testName](QSharedPointer<QObject>& elem)
        { return elem->objectName() == testName; }) == end(m_tests) )
       m_tests.push_back(QSharedPointer<QObject>(test));
}

And the macro turns into the cleaner:

#define DECLARE_TEST(className) static TestAdder<className> t_##className(#className);

By the way, here is how our widget test looks like:

--- WidgetTest.h" ---
#ifndef WIDGETTEST_H
#define WIDGETTEST_H

#include "TestRunner.h"
// Qt includes
#include <QObject>

class WidgetTest : public QObject
{
 Q_OBJECT
private slots:
 void My_first_test();
 void On_button_pressed_should_save_the_world();
};

DECLARE_TEST(WidgetTest)
#endif //WIDGETTEST_H

--- WidgetTest.cpp ---
#include "WidgetTest.h"

void WidgetTest::My_first_test()
{
  ...
}

void WidgetTest::On_button_pressed_should_save_the_world()
{
   ... important test 🙂 ...
}

And main.cpp is straightforward:

#include "TestRunner.h"
int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    return RUN_ALL_TESTS(argc, argv);
}

It’s simple, isn’t it? Now you can add tests easily. Disabling a test? Create an “empty” macro like DECLARE_TEST_DISABLED(className) that does nothing, or just comment the line DECLARE_TEST(className).

The second issue I’m dealing with is about test fixtures: what if you find yourself writing two or more tests that operate on similar data? For example, GTest allows you to reuse the same configuration of objects for several different tests. It is damned shrewd about this and it uses inheritance to make each test function have its own copy of the data. I heartily recommend you to dive into the internals!

Here I’d like to keep on exploiting QTest’s running mechanism (e.g. automatic execution of all the private slots of a test class), maybe adding some tricks to increase our productivity. So, we know QTest treats four slots in a special way:

• initTestCase() will be called before the first testfunction is executed,
• cleanupTestCase() will be called after the last testfunction was executed,
• init() will be called before each testfunction is executed,
• cleanup() will be called after every testfunction,

Quick recap, we have:

#ifndef WIDGETTEST_H
#define WIDGETTEST_H

#include "TestRunner.h"
// Qt includes
#include <QObject>

class WidgetTest : public QObject
{
 Q_OBJECT
private slots:
 void My_first_test();
 void On_button_pressed_should_save_the_world();
private:
 SomeData m_data;
};

DECLARE_TEST(WidgetTest)
#endif //WIDGETTEST_H

And we want m_data to be fresh and renewed before each test function gets executed. In a first attempt we could renew m_data in the init slot – that will be called before each testfunction is executed. Ok this sounds good, but what if SomeData is noncopyable/nonassignable (like QObject is)? A possible solution employs pointers – actually I use a smart pointer to avoid deleting the object by hand:

#ifndef WIDGETTEST_H
#define WIDGETTEST_H

#include "TestRunner.h"
// Qt includes
#include <QObject>
// std includes
#include <memory>

class WidgetTest : public QObject
{
 Q_OBJECT
private slots:
 void init();
 void My_first_test();
 void On_button_pressed_should_save_the_world();
private:
 std::unique_ptr<SomeData> m_data;
};

DECLARE_TEST(WidgetTest)
#endif //WIDGETTEST_H

--- WidgetTest.cpp ---

...

void WidgetTest::init()
{
   m_data.reset( new SomeData(aParameter) );
}

Thus, every time a test function is executed, it will have a “new” m_data. This way you can pretend that QTest has fixtures! Another valuable thing you should reckon in is about inheritance: suppose you have a couple of tests that only use common (not shared!) data (say a couple of widgets initialized in a certain way). You can extract this aspect in a class – such as “SpecialWidgetTestBase” – and make your tests inherit from it:

...

class SpecialWidgetTestBase : public QObject
{
   Q_OBJECT
private slots:
   void init(); // reset the unique_ptrs
protected:
   std::unique_ptr<QWidget> m_specialWidget;
   std::unique_ptr<QTextEdit> m_specialTextEdit;
};

...

class GUIPerformanceTest : public SpecialWidgetTestBase
{
   Q_OBJECT
private slots:
   void On_image_load_benchmark();
   void When_mouse_move_benchmark();
};
DECLARE_TEST(GUIPerformanceTest)

...

class GUIActionsTest : public SpecialWidgetTestBase
{
...
};
DECLARE_TEST(GUIActionsTest)

Both your tests will have base class’ protected data resetted before each test function runs! But if a subclass (say GUIActionTest) had redefined the init slot, the parent class’ one would not have been executed – init is not “recursive”. In this case:

• make the init() function in the base class a public slot,
• call it from the init() of the derived class,
• make other stuff in the init() of the derived class (e.g. other resetting)

The last thing I’m going to handle is very trivial, but often can make your work faster: output. I want QTest to color its results. A nasty red line if a test fails, a gentle green one if a test passes. So, please, it’s your productive programmer speaking: make your output colored, I don’t care how, a stupid post-processing executable suffices:

MyProgram.Tests.exe | MakeMyEyesightHappy.exe

And you can peacefully configure your IDE (or editor) to execute a custom command. For example, under Windows, the stupid program can be easier (and terribly better) than this:

#include <windows.h>
#include <iostream>
#include <string>
#include <regex>

enum Color { blue=1, green, cyan, red, purple, yellow, grey, dgrey, hblue, hgreen, hcyan, hred, hpurple, hyellow, hwhite };

using namespace std;

void SetConsoleColor( Color color )
{
   SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), (WORD) color);
}
WORD GetConsoleColor ()
{
   auto consoleHandle = GetStdHandle(STD_OUTPUT_HANDLE);
   CONSOLE_SCREEN_BUFFER_INFO con_info;
   GetConsoleScreenBufferInfo(consoleHandle, &con_info);
   return con_info.wAttributes;
}

class ColorGuard
{
public:
 ColorGuard() : m_color(GetConsoleColor())
 {}

 ~ColorGuard()
 {
    SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), m_color);
 }

private:
 WORD m_color;
};

int main()
{
  string line;

  regex fail("(FAIL.*)");
  regex pass("(PASS.*)");

  while (getline(cin, line))
  {
    ColorGuard guard;

    if (regex_match(line, fail))
    {
      SetConsoleColor(hred);
    }

    if (regex_match(line, pass))
    {
      SetConsoleColor(hgreen);
    }

    cout << line << endl;
  }

  return 0;
}

And you’ll finally have a colored report! Don’t forget to take care of your output: you can also customize the TestRunner seen before making advanced stuff – for example filtering (QTest already provides some filtering, see here).

Then, you know how to improve your QTest process and productivity. Use a test runner, you’ll focus only on tests and you won’t be distracted from deploying them. If you need test fixtures, use pointers (better if smart!) and identify abstractions if some test classes have common data. At last (but not least), don’t ignore the output: the more it is evident, the more fast and productive you become!