How to do softassertion with karate?

Question

I have a feature that use other two features something like that:

When call read(ser.getCarList) headers
When call read(ser.getTaxes) headers

So the first feature getCarList has two validations

When method Get
 * configure continueOnStepFailure = true
Then status 200
And match response = read ('this:getCarAssertion')
 * configure continueOnStepFailure = true

I have tried with the new keyword but when I get a status code 200 but a bad response the next feature getTaxes does not continue in the execution

it is somewhat experimental. I'll ask someone who knows about it to comment here. meanwhile my suggestion is that you should not look at it as a `blocker` and make progress with your evaluation and test-suites. and if this is urgent, please follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue — Peter Thomas, Mar 21 '21 at 08:09

score 5 · Accepted Answer · edited Feb 11 '23 at 19:51

The continueOnStepFailure is a new keyword that was meant to be used when looking to validate results and not fail immediately on the first failure. The purpose was for assertions or validations so as much information can be validated when asserting results of tests.

To avoid its usage to be as a pure if condition for several steps (with unexpected consequences), the default behavior of * configure continueOnStepFailure = true will only continue the execute if the fails happen in a match step and once you disable the mechanism with * configure continueOnStepFailure = false the test will fail (but still provide details for each step within the continueOnStepFailure block). This is because match is the recommended keyword for any sort of validations and is how you can leverage the powerful JSON assertions library etc.

It is also recommended to also explicity set * configure continueOnStepFailure = false after the set of match keywords so there are no unexpected behaviors after that conscious decision of continuing to evaluate keywords after a failure.

That being said there are ways to extend and configure the behavior of the continueOnStepFailure beyond the default behavior. The keyword also takes a JSON input, instead of a boolean, that allows some more extensibility. E.g. the default behavior of the keyword can be represented as follows:

* configure continueOnStepFailure = { enabled: true, continueAfter: false, keywords: ['match'] }

This means the continueOnStepFailure mechanism will be enabled, the scenario execution will not continue after the mechanism is disabled and it'll only accept failures if those happen in the match keyword. Note that if you set continueAfter to true the scenario will continue to execute the remaining steps but the scenario itself will still be marked as failed (with appropriate output in the report and typical failed behavior for any caller of that scenario). I highly discourage to set continueAfter to true.

For your specific use case, the status keyword is definitely within the boundaries of assertions that I've described. status 200 is just a shortcut for match responseStatus == 200. Very likely we should add status to the default behavior, given that it's a match assertion. With the extended configuration in JSON you can do the following for your use-case:

When method Get
And configure continueOnStepFailure = { enabled: true, continueAfter: false, keywords: ['match', 'status'] }
Then status 200
And match response = read ('this:getCarAssertion')
And configure continueOnStepFailure = false

Some additional examples can be found in the unit tests in this pull request. For quick reference, this is how your Karate Test report will look like:

How to do softassertion with karate?

1 Answers1

Linked