Should I test all enum values in a contract?

Question

I have a doubt about about whether I should consider a certain type of test functional or contract.

Let's say I have an API like /getToolType, that accepts a {object" "myObject"} as input, and returns at type in the form {type: "[a-z]+"}

It was agreed between client and server that the types returned will match a set of strings, let's say [hammer|knife|screwdriver], so the consumer decided to parse them in an enum, with a fallback value when the returned type is unknown.

Should the consumer include a test case for each type(hammer, knife, screwdriver) to ensure the producer is still following the agreement that it will always return , for instance , the lowercase string "hammer" when /getToolType is called with an hammer object? Or would you consider such a test case as functional? And why?

Answer 1

IMO the short answer is 'no'.

Contract testing is more interested in structure, if we start boundary testing the API we move into functional test territory, which is best done in the provider code base. You can use a matcher to ensure only one of those three values is returned, this should ensure the Provider build can't return other values.

I would echo @J_A_X's comments - there is no right or wrong answer, just be wary of testing all permutations of input/output data.

Answer 2

Great question. Short answer: there's no right or wrong way, just how you want to do it.

Longer answer:

The point of Pact (and contract testing) is to test specific scenarios and making sure that they match up. You could simply, in your contract, create a regex that allows any string type for those enums, or maybe null, but only if your consumer simply doesn't care about that value. For instance, if the tool type had a brand, I wouldn't care about the brand, just that it's returned back as a string since I just display the brand verbatim on the consumer (front-end).

However, if it was up to me, from what I understand of your scenario, it seems like the tool type is actually pretty important considering the endpoint it's hitting, hence I would probably have specific tests and contracts for each enum to make sure that those particular scenarios on my consumer are valid (I call X with something and I expect Y to have tool type Z).

Both of these solutions are valid, what it comes down to is this: Do you think the specific tool type is important to the consumer? If it is, create contracts specific to it, if not, then just create a generic contract.

Hope that helps.

Answer 3

The proper state is that consumer consumes hammer, knife, and screwdriver, c=(hammer,knife,screwdriver) for short while producer produces hammer, knife, and screwdriver, p=(hammer,knife,screwdriver). There are four regression scenarios:

1. c=(hammer,knife,screwdriver,sword), p=(hammer,knife,screwdriver)
2. c=(hammer,knife,screwdriver), p=(hammer,knife,screwdriver,sword)
3. c=(hammer,knife,screwdriver), p=(hammer,knife)
4. c=(hammer,knife), p=(hammer,knife,screwdriver)

1 and 3 break the contract in a very soft way. In the 1st scenario, the customer declared a new type that is not (yet) supported by the producer. In the 3rd scenario, the producer stops supporting a type. The gravity of scenarios may of course wary, as something I consider soft regression, might be in a certain service in a business-critical process. However, if it is critical then there is a significant motivation to cover it with a dedicated test case. 2nd and 4th scenarios are more severe, in both cases, the consumer may end up in an error, e.g. might be not able to deserialize the data.

Having a test case for each type should detect scenario 3 and 4. In the 1st scenario, it may trigger the developer to create an extra test case that will fail on the producer site. However, the test cases are helpless against the 2nd scenario. So despite the relatively high cost, this strategy does not provide us with full test coverage.

Having one test case with a regex covering all valid types (i.e. hammer|knife|screwdriver) should be a strong trigger for the consumer developer to redesign the test case in 1st and 4th scenario. Once the regex is adjusted to new consumer capabilities it can detect scenario 4 with probability p=1/3 (i.e. the test will fail if the producer selected screwdriver as sample value). Even without regex adjustment, it will detect the 3rd scenario with p=1/3. This strategy is helpless against the 1st and 2nd scenario.

However, on top of the regex, we can do more. Namely, we can design the producer test case with random data. Assuming that the type in question is defined as follows:

enum Tool {hammer,knife,screwdriver}

we can render the test data with:

responseBody = Arranger.some(Tool.class);

This piece of code uses test-arranger, but there are other libraries that can do the same as well. It selects one of the valid enum values. Each time it can be a different one. What does it change? Now we can detect the 2nd scenario and after regex adjustment the 4th one. So it covers the most severe scenarios. There is also a drawback to consider. The producer test is nondeterministic, depending on the drawn value it can either succeed or fail which is considered to be an antipattern. When some tests sometimes fail despite the tested code being correct, people start to ignore the results of the tests. Please note that producer test case with random data is not the case, it is in fact the opposite. It can sometimes succeed despite the tested code is not correct. It still is far from perfect, but it is an interesting tradeoff as it is the first strategy that managed to address the very severe 2nd scenario.

My recommendation is to use the producer test case with random data supported with a regex on the customer side. Nonetheless, there is no perfect solution, and you should always consider what is important for your services. Specifically, if the consumer can safely ignore unknown values, the recommended approach might be not a perfect fit.