0

We have a CI system using TFS 2015, and all was working well.
However, we added some new tests, and the "Gather Artifacts" stage stopped working.
I tracked this down to the fact that the MSTest command-line gets built thus:

MsTest.exe /testcontainer:blah /test:test1 /test:test2 [...] /resultsfile:"c:\blah.trx"

So when we go above a certain number of tests, the command-line ends up too long and becomes truncated, losing the /resultsfile switch and possibly some of the /test: switches also.

In this instance, TFS reports the suite as having passed, despite the fact that all tests did not run.

I see in the documentation for MSTest.exe that there is a /testlist: parameter that lets you specify which tests run by pointing at a file.

Can anyone help me work out how I switch over to using a testlist, or some other way of solving this issue?

Clive Galway
  • 482
  • 4
  • 13
  • Is there a reason to use MSTest? We had the same problem and swapped to VS Test, MS Test is "legacy" since 2010 IIRC. The other alternative is to amalgamate test projects to reduce the number you have and use shorter assembly names. – DaveShaw Jan 19 '17 at 12:43
  • Unfortunately, I do not know a whole lot about this system, so am unsure if we can use VSTest. Trying to shorten the assembly names etc is not a solution in my book, as you will always end up running into this problem if you have enough tests. Is this fundamentally an unsolvable problem if using MSTest? – Clive Galway Jan 19 '17 at 12:45
  • Also, is VSTest not for Visual Studio only? The build vms do not have Visual Studio on them... – Clive Galway Jan 19 '17 at 12:48
  • The limitation is not that of MSTest, but of the [Windows Command Line AFAIK](http://stackoverflow.com/questions/3205027/maximum-length-of-command-line-string) - the problem is with MSTest all test assembly names are passed via Command Line args, with VS Test, it is passed another way that doesn't have an upper limit. I'd suggest you look at VS Test and putting VS on the build servers, even [MS does it](https://www.visualstudio.com/en-us/docs/build/admin/agents/hosted-pool#software-on-the-hosted-build-server). – DaveShaw Jan 19 '17 at 22:45

2 Answers2

1

I found the solution, it was to switch from using "Test Plan" to "Test Assembly"

example

Clive Galway
  • 482
  • 4
  • 13
0

Just as DaveShaw said in the comment, the limitation is not of MSTest or TFS, it's related to Windows Command Line AFAIK.

We can use both VSTEST and MSTEST to run automated unit and coded UI tests from a command line.

  • VSTest.Console.exe is optimized for performance and is used in place of MSTest.exe in Visual Studio.

After finish running the test. It won’t save the results and there is no .trx file generated in the target directory.

  • MSTest.exe You can use the MSTest.exe program to run automated tests in a test assembly from a command line.

MSTest is used for load tests and for compatibility with Visual Studio 2010 test projects.

MSTest can also be used to view the test results from these test runs, save the results to disk, and save your results to Team Foundation Server. More detail info you could refer this link: How to choose between tcm.exe, mstest.exe and vstest.console.exe

Since you are using TFS2015, and if you don't have any VS2010 test projects, we encourage you to use VStest instead of MStest.

Community
  • 1
  • 1
PatrickLu-MSFT
  • 49,478
  • 5
  • 35
  • 62
  • TFS does not feature an option for VSTest I am not talking about running tests manually from the command-line, I am talking about when TFS runs the tests using CI. – Clive Galway Feb 10 '17 at 16:32