2

We have a cluster where only a single server (master) can return a meaningful result. The master is selected automatically and at test time I don't know who is master and who is not. In our tests we are using testinfra which gives us a fixture 'host' for each host in cluster.

We have a test which looks like this:

def test_cluster(host):
    assert host.run_command('cluster --is_ok').stdout == 'OK'

The problem is that there are N hosts, and only one of them can return OK, others says "I'm not a master".

The normal output looks like this:

test_cluster[host1] .... FAIL
test_cluster[host2] .... OK
test_cluster[host3] .... FAIL
test_cluster[host4] .... FAIL

I want to find a way to mark the test test_cluster as pass if any of tests is pass (and fail if all tests are failed).

If testinfra is too much for the test, here the more concise example with pytest only:

@pytest.fixture.parametrize("num", [0, 0, 1, 0])
def test_example(num):
     asssert num > 0

I need pytest to pass if there is at least one 1 in 'num' fixture.

George Shuklin
  • 6,952
  • 10
  • 39
  • 80
  • If some test is expected to fail, mark it with `@pytest.mark.xfail`. – hoefling Jul 08 '20 at 07:52
  • 2
    `@pytest.mark.xfail` marks all instances of the test, and I need to fail if all instances failed. – George Shuklin Jul 10 '20 at 13:52
  • I see. What do you expect the custom behaviour to be, exit with zero if any of `test_cluster` tests passes? – hoefling Jul 10 '20 at 14:04
  • If all instances of the test failed -> test failed. If at least one from parametrization passes, then the test pass. (It's not about generic 'all tests' it's about tests after parametrization within parametrization group). – George Shuklin Jul 13 '20 at 15:26
  • That's not how parametrize should be used, those aren't separate test cases with the same expected outcome; the correct test for your second example would be something like `assert any(num > 0 for num in [0, 0, 1, 0])`. – jonrsharpe Jul 15 '20 at 09:44
  • In my case the fixture for parametrization is `host` (from testinfra) and it represents different servers (provided by Ansible inventory). I can do my test with custom code to connect to all servers in sequence, but it's very much an anti-pattern for testinfra, so I would prefer to have a way to have fixture-wide assert (or cross-test assert? How should I call it?). – George Shuklin Jul 15 '20 at 10:11

0 Answers0