2

We are using TeamCity for Selenium tests execution with Specflow+Specrun, the problem is that TeamCity count re-executed tests.

For example if some test failed for first time he will be re-executed two times more, in teamcity we will see that three tests failed, but probably it was one test.

Also if first re-execution will fail, but other two will success, this will be reported in teamcity as two failed, one passed, but I need to be reported that only one test has been passed.

It is possible to configure this in TeamCity using service messages or something else?

UPDATED:

Based on answer we can gather logs using powershell script and change build status using teamcity service messages:

$path = Get-ChildItem %teamcity.build.checkoutDir%\ProjectFolder\bin\Remote\TestResults\specrun.log
$file = Get-content $path
$total = $file | select-string "Total:"
$passed = $file | select-string "Succeeded:"
$failed = $file | select-string "Failed:"

write-host $( "##teamcity[buildStatus text='Tests {0}, {1}, {2}']" -f $total, $passed, $failed )
Shilovskoye Moloko
  • 93
  • 1
  • 2
  • 14

1 Answers1

1

TeamCity is behaving as expected. TeamCity reports what Testing Framework tells him. Roughly the process is:

  • Execute step that run tests.
  • Gather the output logs.
  • Extract stats: number of test executed, passed and failing.
  • Fail the build if tests fail if configured to do that.

You have to configure your Testing Framework (specflow+specrun in your case) to change the way it reports the re-execution. You should configure specflow to log the re-executions in a different way... if it can.

In general you should avoid re-execution of tests. The tests should work at the first attempt. They should prepare the state if needed, do their test, and clean the state.

The need of re-execution indicates probably some dependency with the other tests. Something like:

  • Test 001: Search for user Foo1. (FAILS)
  • Test 002: Adds user Foo1. (OK)
  • Re-execution of Test 001: Search for user Foo1. (OK)

The correct way is that TEST001 does not have a dependency of TEST002, adds the user Foo1, test search and clean user Foo1.

Oscar Foley
  • 6,817
  • 8
  • 57
  • 90