23

On Bamboo I have MSBuild job for building and running tests.

It works fine - but I want to use test quarantine option so when I have some test moved to quarantine (and this moved tests are only what fail) job will be marked as successful

I moved one failing test to quarantine but job is marked as failed

Configuration:

In Job -> Tasks I have Command named Test runner:

Executable = VSTest.Console

Argument = "WebServiceTestClient.dll" /Logger:trx

And after running job with failing test quarantined I get log

01-Apr-2014 10:59:44  Total tests: 62. Passed: 61. Failed: 1. Skipped: 0. 
01-Apr-2014 10:59:44  Test Run Failed. 
01-Apr-2014 10:59:44  Test execution time: 10.1296 Minutes 
01-Apr-2014 10:59:44  Failing task since return code of [C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\VSTest.Console.exe "WebServiceTestClient.dll" /Logger:trx] was 1 while expected 0 
01-Apr-2014 10:59:44  Finished task 'Test runner' 
01-Apr-2014 10:59:44  Starting task 'Parse test logs' of type 'com.atlassian.bamboo.plugin.dotnet:mstest' 
01-Apr-2014 10:59:44  Parsing test results... 
01-Apr-2014 10:59:45  Failing task since 1 failing test cases were found.

Job is marked failed but then:

01-Apr-2014 10:59:45  Changing Task Result to SUCCESS as all failed tests were quarantined. 
01-Apr-2014 10:59:45  Finished task 'Parse test logs' 
01-Apr-2014 10:59:45  Running post build plugin 'NCover Results Collector' 
01-Apr-2014 10:59:45  Running post build plugin 'Clover Results Collector' 
01-Apr-2014 10:59:45  Running post build plugin 'Artifact Copier' 
01-Apr-2014 10:59:45  Finalising the build...

Unfortunately the output is: enter image description here

pbaranski
  • 22,778
  • 19
  • 100
  • 117
  • 9
    Bambo's treating all return other than 0 as failure while usually a program return 1 when there is a warning (or the like). So my work around is to use an inline script task (instead of command task), in the task first run vstest.console then catch the return value (`%ERRORLEVEL%`) if it's 1 then do `EXIT /B 0` – tmlai May 10 '14 at 05:33
  • Thank ou tmlai... Your comment is actually the answer. I created a powershell script doing the tests and added a new task to parse the mstest trx. – Nordes Jul 16 '15 at 08:31

1 Answers1

1

The problem is the job result is a failure.

You can run the tests via an ant script and as long as the target executes successfully you will pass. In my bamboo setup I run all the unit tests like this and after running the test I look for the test results and generate a failed test result if one is missing (for ex, if the cppunit exe threw an exception and didn't write out its results).

Then the final stage is set to check for the test results (this can be the same stage you run the tests from or a later one).

BoldAsLove
  • 660
  • 4
  • 13