QT function suppressing test failures in python

Question

Working on a complex unittest that is testing a python UI, and a QT function appears to be suppressing test failures. I believe I've been able to create a minimal file that repeats the behavior:

import pytest
from unittest import TestCase
from PySide2.QtCore import QTimer

def meaningless():
    return 'fire'


class TestClass(TestCase):

    def test_1(self):
        def inner_test_1():
            self.assertEqual(meaningless(),'x')
        
        inner_test_1()

    def test_2(self):
        def inner_test_2():
            self.assertEqual(meaningless(),'x')

        QTimer.singleShot(1, inner_test_2)


if __name__ == '__main__':
    import sys

    sys.exit(pytest.main([__file__]))

The first test fails as it should, but the second passes erroneously. In my more complex real-world unit test, the equivalent of "inner_test_2" does indeed fire, and the resulting assertion error can be seen in the test logs, but the test does not register as a failure. From what I can tell in the QT docs, this may have something to do with multi-threading? How do I get failures to fail?

Your example isn't running an event-loop, so the behaviour is correct and as expected. You should probably be usng something like [pytest-qt](https://pypi.org/project/pytest-qt/). — ekhumoro, Nov 24 '21 at 14:24
Sorry if I'm splitting hairs here, but in this case "expected behavior" = "not working because it's being used wrong" or "this is a legitimate use of QTimer, but it won't produce what you're looking for." ? — Jed Godsey, Nov 24 '21 at 17:07
Well, you wrote the test-case, so only you can really answer that. Are you testing for success or failure? What exactly do you expect a [single-shot timer](https://doc.qt.io/qt-5/qtimer.html#singleShot) to do, if there's no running event-loop? Test-cases are partly (or sometimes even mainly) intended to document the programmer's understanding of an API, rather than just the API itself. — ekhumoro, Nov 24 '21 at 20:42
Since the example you gave is not representative for your real code (your real code executes `inner_test_2` while the example does not) I can only guess that it might be related to multi-threading? It could be a problem if the `assertEqual(meaningless(),'x')` is run in a different thread. — SebDieBln, Dec 01 '21 at 11:58

score 0 · Answer 1 · answered Dec 01 '21 at 21:14

This may have something to do with multi-threading?

Yes, exactly. Failing tests are detected via exceptions and exceptions in a different thread are not caught by the unittest framework, because it can not associate a thread with a test.

A minimal, reproducible example would be:

import pytest
from unittest import TestCase
import threading

class TestClass(TestCase):
    def test_2(self):
        def inner_test_2():
            self.assertTrue(False)
        myTestThread = threading.Thread(target=inner_test_2)
        myTestThread.start()
        myTestThread.join()

if __name__ == '__main__':
    import sys

    sys.exit(pytest.main([__file__]))

Note that while assertTrue(False) raises an exception we do not see a failing test. But what we do see (besides the stacktrace of the exception) is a warning from pytest:

    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/warnings.html

How do I get failures to fail?

You need to catch the exceptions from the thread and raise them in the main thread so the unittest framework can pick them up and understand what test failed. How to do that is a different topic, see Make Python unittest fail on exception from any thread.

QT function suppressing test failures in python

1 Answers1