All tests pass but TFS marks the build as partially succeded

Question

Our current project involves building a robotized box controlled by a .Net application. We interface with quite a few hardware libraries and we did set up a integration server with all the hardware connected to it to run nightly regression tests.

Unfortunately, not all hardware libraires in our setup integrates nicely with TFS and MSTest.

When we run our build and tests with a certain librairy, we have either one of these 2 behaviors:

All tests are martked as passed in TFS but the build is shown as partially succeeded.
TFS shows no test run, but the log shows that all tests passed, build is marked as partially succeeede.

Looking at the logs, we can see these 2 lines after the tests are published to the TFS server, "Handle MSTest Exception" "MSTest.exe returned an exit code of 0 indicating that not all tests passed"

What really puzzles me is that running the same tests with mstest from the command line does not display this problem. Futhermore, when running from the command line, MsTest returns 0 when all tests pass and 1 when there is an error

So my questions are:

What is the appropriate MSTest return code on Success/Failure
Beside the log in visual studio, is there any other detailed loggin feature available?
Any clue on where to look?

Additional info: We did notice a difference when using (or not) the "NoIsolation" flag in mstest. The specific tests will pass when using that flag, however other tests will fail... We are still investigating that one.

Thanks

EDIT: The relevant portion of the log: 30/30 test(s) Passed

Summary
-------
Test Run Completed.
Passed  30
----------
Total   30
Results file:  C:\Builds\1\Galil Daily build\TestResults\D201364-W7$_D201364-W7 2011-07-05 10_23_33_Any CPU_Debug.trx
Test Settings: Default Test Settings
Waiting to publish...
Publishing results of test run D201364-W7$@D201364-W7 2011-07-05 10:23:33_Any CPU_Debug to http://mtlapp07:8080/tfs/DI_DEV...
..Publish completed successfully.
Final Property Values
Category = Galil
CommandLineArguments = /noisolation
Flavor = 
MaxPriority = -1
MinPriority = -1
PathToResultsFilesRoot = C:\Builds\1\Galil Daily build\TestResults
Platform = 
Publish = True
SearchPathRoot = C:\Builds\1\Galil Daily build\Binaries
TestConfigId = -1
TestConfigName = 
TestContainers = System.Linq.OrderedEnumerable`2[System.String,System.String]
TestLists = 
TestMetadata = 
TestNames = 
TestSettings = 
ToolPath = 
Version = -1
Final Property Values
Condition = False
Final Property Values
Condition = True

00:00
Handle MSTest Exception
 MSTest.exe returned an exit code of 0 indicating that not all tests passed.

00:00
If testException is NOT TestFailureException
Initial Property Values
Condition = False
Final Property Values
Condition = False

Edit 2.0:

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Galil;

namespace TestProject1
{
[TestClass]
public class UnitTest1
  {
    [TestMethod]
    public void TestMethod1()
    {
        Galil.Galil m_galil = new Galil.Galil();
        m_galil.address = "COM4 19200";
        string resp = m_galil.command("MTA=2.5;");
        Assert.AreEqual(":", resp);
        try
        {
            m_galil.address = "OFFLINE";
        }
        catch (Exception)
        {
        }
    }

    [TestMethod]
    public void TestMethod2()
    {
        Galil.Galil m_galil = new Galil.Galil();
        m_galil.address = "COM4 19200";
        string resp = m_galil.command("MTA=2.0;");
        Assert.AreEqual(":", resp);
        try
        {
            m_galil.address = "OFFLINE";
        }
        catch (Exception)
        {
        }
    }
  }
}

Can you post your logs please? – Mike Veigel Jul 05 '11 at 17:22 — Mike Veigel, Jul 05 '11 at 17:22

score 2 · Accepted Answer · answered Jul 05 '11 at 17:36

2

What is the appropriate MSTest return code on Success/Failure

Exit code 1 = not all tests pass

Example:

C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(1377,5,1377,5): warning : MSTest.exe returned an exit code of 1 indicating that not all tests passed.

All tests pass

Example:

Waiting to publish... Publishing results of test run Company_TFSBuild_SVC@BUILD-DEV 2011-07-01 13:15:46_Any CPU_Release to http://company-source:8080/...
..Publish completed successfully.

Beside the log in visual studio, is there any other detailed loggin feature available?

You can get a more detailed log by going to:

Menu Bar -> Tools -> Options... -> Projects and Solutions -> MSBuild project build output verbosity

Any clue on where to look?

Need to see your log file.

answered Jul 05 '11 at 17:36

Mike Veigel

3,795
2
20
28

The relevant portion of the log was added – Vincent Hubert Jul 05 '11 at 18:13
Is MsBuild also used to run tests on VS2010? It seems that it uses a xml build definition file that does not look like a MSBuild build script. – Vincent Hubert Jul 05 '11 at 18:23
A wrapper is created to invoke MSTest.exe inside MSBuild. MSTest.exe runs the tests. – Mike Veigel Jul 05 '11 at 18:47
Are you using asserts inside your tests? If you were to comment out say 29 of the 30 test and run the tests do you still see this message? If so, can you post the code that execs that unit test? – Mike Veigel Jul 05 '11 at 19:00
We do use asserts inside our tests. I have not tried commenting out tests on the buid server but I suspect running only one test could work: on my machine, when I run on the console, one of the library spits out error messages on stderr. Could that cause MsBuild / TFS build server to flag something as failed? – Vincent Hubert Jul 05 '11 at 19:10
I would have to see your unit test suite to track down the issue. If I were you, based on what I see in the log (assuming you posted the complete diagnostic log and not minimal or normal) would be to comment out unit tests one by one and find out what, if any, are causing it to fail. If they all fail then we have a different issue and can take it from there. – Mike Veigel Jul 05 '11 at 19:21
Write your catch blocks like this: catch{ Assert.Fail(); } If you want to react to those failing. Also, did you try using just one unit test? Did you experience that issue still? – Mike Veigel Jul 06 '11 at 19:44
We do not want to react to the failings. THe library we use throws an exception when we disconnect. In that case, it can be safely ignored. Using only one test will work. We suspect a static variable or threading issue in the componenent we use. Thanks for your help. – Vincent Hubert Jul 07 '11 at 18:31
You are welcome and I am sure you will figure out where the problem is in no time troubleshooting in this manner. – Mike Veigel Jul 07 '11 at 18:46

score 2 · Answer 2 · answered Sep 16 '11 at 19:33

2

Actually, we were able to isolate the problem to a library that did output strings into the standard error stream. Somehow, if TFS sees something on the standard error, it will flag the test as partially completed.

answered Sep 16 '11 at 19:33

Vincent Hubert

1,386
9
23

I had this issue. Was passing invalid command line arguments for some reason, MSBuild didn't like that and logged it, step got marked as partially succeeded. – Edgar May 18 '17 at 12:12

All tests pass but TFS marks the build as partially succeded

2 Answers2

Linked