0

Generally when I use Cucumber and other tools to drive a browser and do end-to-end tests I consider noise in the suite (which we'll loosely define as failing tests that may not actually point to problems in app code programming) to be normal, and the suites are triaged by our team with the help of a custom tool we've made and maintain for that purpose.

I want to consider a new type of suite that we run, that passes if one important final assertion is met per test and marks a result as inconclusive (or pending is fine) if environmental issues or problems with the test code itself make it so that a test fails. Failures in the test suite should only occur if a test makes it to the final assertion (the point of each smoke test) and fails to meet expectations.

This effort, which is different than what our team normally does, prompts us also to consider that we should log 'warnings' along the way, things we would normally assert with Rspec expectations and have fail, with the key difference of now not throwing an exception, and instead having a log that inspires and informs exploratory testing.

So, a set of bullet pointed questions for the impatient:

  • How can I make sure certain exceptions are turned into pending results instead? My idea is something similar to this in the hooks

    Around('@end_to_end_ci') do |scenario, block|
  begin
    block.call
  rescue Watir::Exception, PageObject::Exception => e #will this rescue all Watir and pageobject exceptions? Add more as needed...
    pending("Test results inconclusive, non-buildbreaking error caught.\nStack trace: #{e}")
  end
end

Thanks!

Please note that I am not simply asking how to make a cucumber scenario return pending. I am speaking to a more general need to only fail in build-breaking situations and have the ability to choose what assertions and exceptions are counted as failures and which ones are counted as warnings.

David West
  • 2,256
  • 6
  • 32
  • 62
  • Possible duplicate of [How do you mark a Cucumber Scenario as Pending](http://stackoverflow.com/questions/3064078/how-do-you-mark-a-cucumber-scenario-as-pending) – max pleaner Mar 14 '17 at 16:47
  • @maxple I've explained why this is different. See the note at the end of my post. – David West Mar 15 '17 at 16:40

0 Answers0