How can I immediately fail an auto test if a warning occurs

Question

I want to instantly fail any auto test when a warning is printed in the tested code, so that I don't miss it in the output and end up with strange test failures later on.

I thought I could use qInstallMessageHandler() for this. I modified the example from here:

#include <QtTest>

class AutoTest : public QObject
{
    Q_OBJECT

public:
    AutoTest();
    ~AutoTest();

private slots:
    void test_case1();

};

void warningMessageHandler(QtMsgType type, const QMessageLogContext &context, const QString &msg)
{
    QByteArray localMsg = msg.toLocal8Bit();
    const char *file = context.file ? context.file : "";
    const char *function = context.function ? context.function : "";
    switch (type) {
    case QtDebugMsg:
        fprintf(stderr, "Debug: %s (%s:%u, %s)\n", localMsg.constData(), file, context.line, function);
        break;
    case QtInfoMsg:
        fprintf(stderr, "Info: %s (%s:%u, %s)\n", localMsg.constData(), file, context.line, function);
        break;
    case QtWarningMsg:
        fprintf(stderr, "Warning: %s (%s:%u, %s)\n", localMsg.constData(), file, context.line, function);
        exit(1);
    case QtCriticalMsg:
        fprintf(stderr, "Critical: %s (%s:%u, %s)\n", localMsg.constData(), file, context.line, function);
        break;
    case QtFatalMsg:
        fprintf(stderr, "Fatal: %s (%s:%u, %s)\n", localMsg.constData(), file, context.line, function);
        break;
    }
}

AutoTest::AutoTest()
{
    qInstallMessageHandler(warningMessageHandler);
}

AutoTest::~AutoTest()
{

}

void AutoTest::test_case1()
{
    qWarning() << "This should cause the application to exit with code 1";
}

QTEST_APPLESS_MAIN(AutoTest)

#include "tst_autotest.moc"

However, the test application doesn't exit early as expected:

17:32:04: Starting /Users/mitch/dev/temp/autotest-qt5_14_fw-Debug/autotest.app/Contents/MacOS/autotest ...
********* Start testing of AutoTest *********
Config: Using QtTest library 5.14.0, Qt 5.14.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 10.0.0 (clang-1000.11.45.2) (Apple))
PASS   : AutoTest::initTestCase()
QWARN  : AutoTest::test_case1() This should cause the application to exit
PASS   : AutoTest::test_case1()
PASS   : AutoTest::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 2ms
********* Finished testing of AutoTest *********
17:32:04: /Users/mitch/dev/temp/autotest-qt5_14_fw-Debug/autotest.app/Contents/MacOS/autotest exited with code 0

I put a breakpoint in the message handler function and it wasn't hit once. What's going on?

score 2 · Accepted Answer · answered Nov 28 '19 at 16:50

Although the example in the documentation shows the handler being installed at the beginning of main(), this is not sufficient for an auto test. It seems that there are other message handlers that Qt itself installs during the lifetime of an auto test, and these are installed quite a bit later than the constructor of the test. You can verify this yourself if you built Qt from source by putting a breakpoint here.

Qt's test logging infrastructure installs its own message handler as soon as the test itself is created (within the QTEST_APPLESS_MAIN macro).

If you're using QML debugging in Creator, then that will also install its own message handler.

To get around this, install your handler as late as possible:

#include <QtTest>

class AutoTest : public QObject
{
    Q_OBJECT

public:
    AutoTest();
    ~AutoTest();

private slots:
    void initTestCase();
    void test_case1();

};

static QtMessageHandler oldMessageHandler;

// Exits on warnings
void warningMessageHandler(QtMsgType type, const QMessageLogContext &context, const QString &msg)
{
    QByteArray localMsg = msg.toLocal8Bit();
    const char *file = context.file ? context.file : "";
    const char *function = context.function ? context.function : "";
    switch (type) {
    case QtWarningMsg:
        fprintf(stderr, "EXITING - TEST FAILED DUE TO WARNING: %s (%s:%u, %s)\n", localMsg.constData(), file, context.line, function);
        exit(1);
    default:
        oldMessageHandler(type, context, msg);
    }
}

AutoTest::AutoTest()
{
}

AutoTest::~AutoTest()
{

}

void AutoTest::initTestCase()
{
    oldMessageHandler = qInstallMessageHandler(warningMessageHandler);
}

void AutoTest::test_case1()
{
    qWarning() << "This should cause the application to exit with code 1";
}

QTEST_APPLESS_MAIN(AutoTest)

#include "tst_autotest.moc"

Be sure to call the previously installed message handler so that you don't lose any of the functionality that Qt's logging provides.

This outputs:

17:44:44: Starting /Users/mitch/dev/temp/autotest-qt5_14_fw-Debug/autotest.app/Contents/MacOS/autotest ...
********* Start testing of AutoTest *********
Config: Using QtTest library 5.14.0, Qt 5.14.0 (x86_64-little_endian-lp64 shared (dynamic) debug build; by Clang 10.0.0 (clang-1000.11.45.2) (Apple))
EXITING - TEST FAILED DUE TO WARNING: This should cause the application to exit with code 1 (../autotest/tst_autotest.cpp:50, void AutoTest::test_case1())
PASS   : AutoTest::initTestCase()
17:44:44: /Users/mitch/dev/temp/autotest-qt5_14_fw-Debug/autotest.app/Contents/MacOS/autotest exited with code 1

How can I immediately fail an auto test if a warning occurs

1 Answers1