How to define multiple matching patterns in AzureDevOps Pipeline VSTest@2?

Question

Currently I'm trying to set up a new pipeline for our solution and can't get the Visual Studio Test to find the correct set of tests within my solution. Either it picks a DLL that doesn't contain any tests (which leads to a fail of the task) or if I specify the testAssemblyVer2 property it produces a warning that it couldn't find any assembly to test.

The base task configuration we are working with:

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    searchFolder: '$(System.DefaultWorkingDirectory)'
    runInParallel: true
    codeCoverageEnabled: true
    diagnosticsEnabled: true

If we run this, we can see in the output the following configuration (part):

 ...
 Test assemblies : **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**
 ...
 ======================================================
 [command]"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" @d:\a\_temp\66884a11-77b3-11e9-b7cb-25533524cce5.txt
 Microsoft (R) Test Execution Command Line Tool Version 16.0.1
 Copyright (c) Microsoft Corporation.  All rights reserved.

 "d:\a\1\s\Tests\Api\FirstController.Tests\bin\Release\netcoreapp2.1\FirstController.Tests.dll"
 "d:\a\1\s\Tests\Api\SecondController.Tests\bin\Release\netcoreapp2.1\SecondController.Tests.dll"
 "d:\a\1\s\Tests\CreateTranslateStringsFromDeviceConfigurationSettings\bin\Release\netcoreapp2.1\CreateTranslateStringsFromDeviceConfigurationSettings.dll"
 "d:\a\1\s\Tests\Api\FourthController.Tests\bin\Release\netcoreapp2.1\FourthController.Tests.dll"
 "d:\a\1\s\Tests\Api\FifthController.Tests\bin\Release\netcoreapp2.1\FifthController.Tests.dll"
 /Settings:"d:\a\_temp\69a604d0-77b3-11e9-b7cb-25533524cce5.runsettings"
 /EnableCodeCoverage
 /logger:"trx"
 /TestAdapterPath:"d:\a\1\s"
 Starting test execution, please wait...

As you can see, there is one assembly CreateTranslateStringsFromDeviceConfigurationSettings that doesn't contain any tests but is picked as a candidate for tests. I took the exact original name from my concrete solution just to show that it obviously doesn't match the pattern but is picked. Now we try to avoid this problem by defining our own match pattern.

If we create the task through the assistant, it will add the following value by default:

    testAssemblyVer2: '**\*test*.dll
      !**\*TestAdapter.dll
      !**\obj\**'

If we run this we get the following output:

...
 Test assemblies : **\*test*.dll !**\*TestAdapter.dll !**\obj\**
...
 ##[warning]No test assemblies found matching the pattern: **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**.

In the output you can see, that the list of test assemblies isn't comma separated, which gives some indication that the value is not understood correctly and therefore maybe leads to the empty list.

So we try to simply copy and paste the comma values from the first running output which produces the following configuration and (failed) output:

    testAssemblyVer2: '**\*test*.dll,!**\*TestAdapter.dll,!**\obj\**'

Output:

 ...
 Test assemblies : **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**
 ...
 ##[warning]No test assemblies found matching the pattern: **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**.

The output matches the first now, but it still doesn't work. So using commas doesn't seem to be the way to go.

So in a fourth case I took the value from the documentation which is

testAssemblyVer2: '**\*test*.dll!**\*TestAdapter.dll!**\obj\**'

But it also failed with the similar error message:

...
Test assemblies : **\*test*.dll!**\*TestAdapter.dll!**\obj\**
...
##[warning]No test assemblies found matching the pattern: **\*test*.dll!**\*TestAdapter.dll!**\obj\**.

So how to define multiple patterns the correct way?

score 16 · Accepted Answer · edited Feb 03 '20 at 15:38

16

Try this:

- task: VSTest@2
  inputs:
    testAssemblyVer2: |
     **\*test.dll
     !**\*TestAdapter.dll
     !**\obj\**
    searchFolder: '$(System.DefaultWorkingDirectory)'

edited Feb 03 '20 at 15:38

dan1st

12,568
8
34
67

answered May 16 '19 at 10:26

Shayki Abramczyk

36,824
16
89
114

1

Yes, that works. There seems to be a bug, if the first pattern is on the same line as the property name. Also Microsoft should adopt the documentation for this. – Oliver May 16 '19 at 11:26
Made a [report to Microsoft](https://developercommunity.visualstudio.com/content/problem/570805/working-with-multiple-pattern-matchings-in-vstest.html). – Oliver May 16 '19 at 11:44
Yea, It look like a bug. Thanks for the report! – Shayki Abramczyk May 16 '19 at 11:48
Thank-you, this helped me figure out why my test assembly was not being found. MS needs better documentation or examples regarding this. – Sharbel Dec 19 '19 at 15:00

How to define multiple matching patterns in AzureDevOps Pipeline VSTest@2?

1 Answers1