Where to add assertion for each generated test by Intellitest

Question

Here I am going to explain the issue by an example. The original question presents the problem more abstractly. No need to read it though.

Update: Question as an example

Lets say we have implemented this buggy function for finding the min of int[]:

public int MyMin(int[] data)
{
    int min = 1000;

    for (int i = 1; i < data.Length; i++)
    {
        if (data[i] < min)
        {
            min = data[i];
        }
    }

    return min;
}

Running Intellitest on this function gives us: enter image description here

Note for test #4 and #6 the function is not calculating the min value correctly due to its buggy implementation. However, these tests are passing which is not desired.

Intellitest cannot magically determine our intended behavior of MyMin and craft the test to fail on these inputs. However, it would be very nice if we could manually specify the desired result for these tests.

@michał-komorowski's solution is feasible, but for each test case I have to repeat its input in terms of PexAssumes. Is there a more elegent/clean way to specify the desired output for the test inputs?

Original Question

Intelitest generates a parameterized test that is modifiable and general/global assertions can be added there. It also generates the minimum number of inputs that maximize the code coverage. Intellitest stores the inputs as individual unit tests, each one calling the parameterized test with a crafted input.

I am looking for a way to add assertion per each input.

As each input is stored as a unit test function in a .g.cs file, the assertion can be added there. The problem is that these functions are not supposed to be customized by the user since they will get overwritten by Intellitest in its subsequent runs.

What is the recommended way of adding assertions for each unit test?

score 2 · Answer 1 · answered Feb 09 '16 at 07:49

You shouldn't add assertions to test methods (methods with [TestMethod] attribute). They are only used to provide parameters values. The place to put assertions are methods with [PexMethod] attribute.

At first glace it may look like a limitation. However, if we consider how IntelliTest works it isn't. There is no sense to add assertion per each input because inputs can be deleted, updated or created at any moment. For example when:

A method being tested was changed.
PexAssume class was used.
Configuration of PexMethod attribute was changed.

However, you can do something else i.e. to add more than one "Pex method" for a method being tested and use PexAssume. For example lets assume that we have BubbleSort method and we want to define different assertions depending on the length of an input array.

[PexMethod]
public void BubbleSort(int[] a)
{
    PexAssume.IsTrue(a.Length == 5);
    int[] result = Program.BubbleSort(a);
    // Assertions specific for an array with 5 elements
}

[PexMethod]
public void BubbleSort(int[] a)
{
    PexAssume.IsTrue(a.Length == 10);
    int[] result = Program.BubbleSort(a);
    // Assertions specific for an array with 10 elements
}

Hi Michal. While this can be helpful to add assertions for general conditions (e.g. array length in your example), but it's cumbersome and repetitive to specify each test input by `PexAssume`. Please take a look at the updated question where I present an example. — Isaac, Feb 09 '16 at 21:54

score 0 · Answer 2 · answered Apr 12 '16 at 17:05

This answer builds on the previous answer. It's more specific for the question that was asked.

Pex generates a test for all code paths, but doesn't know anything about your code. You still have to do arrange/act/assert in the PUT (parameterized unit test) to tell Pex how you think your code should work. Additionally, you can add assume before the arrange so the pattern is assume/arrange/act/assert.

For your example, I start with this PUT.

    [PexMethod(MaxRunsWithoutNewTests = 200)]
    [PexAllowedException(typeof(NullReferenceException))]
    public int  MyMin([PexAssumeUnderTest]Class1 target, int[] data)
    {
        //assume
        PexAssume.IsTrue(data.Length == 1);
        //arrange
        data[0] = 0;

        //act
        int result = target.MyMin(data);

        //assert
        PexAssert.AreEqual(0, result);
        return result;
    }

The results show that only 3/8 blocks were covered and that test 2 failed with Expected '0', got '1000'

This tells me I need to look at the code to figure out why I got 1000.

I see that I'm starting the for loop with 1 instead of 0. So I fix the code and run IntelliTest again.

This time I get two passing tests which is good. But only 6/8 blocks have been tested. I'm missing something.

I create a new PUT which allows Pex to generate the data which looks like this.

    [PexMethod(MaxRunsWithoutNewTests = 200)]
    [PexAllowedException(typeof(NullReferenceException))]
    public int MyMin2([PexAssumeUnderTest]Class1 target, int[] data)
    {
        //assume

        //act
        int result = target.MyMin(data);

        //assert
        return result;
    }

Now I have 7 unit test which exercise all code paths and all tests pass.

You'll notice that each PUT produces its own set of tests.

Where to add assertion for each generated test by Intellitest

Update: Question as an example

Original Question

2 Answers2