VSTest@2 Task is Updating / Overriding coverlet.runsettings and not respecting exclusions

Question

I am trying to add code coverage to a .NET Framework 4.8 solution in Azure DevOps. I have based my approach on the questions here and here and the coverlet docs, and am successfully getting code coverage results.

However this solution also contains a number of shared projects which have their unit tests in a different solution. I wish to exclude these projects, and the test projects themselves from the coverage report.

In my coverlet.runsettings file I have included the following lines based on the docs:

<Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*,[Company.*UnitTest]*</Exclude>
<Include>[Company.Application.*]*</Include>

However when the tests are run using the VSTest@2 I see the following in the logs

Provided settings file:
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>          
          <Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*,[Company.*UnitTests]*</Exclude>
          <Include>[Company.Application.*]*</Include>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>
Updated Run Settings:
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <!-- As above -->
      </DataCollector>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <!-- Lots of configuration ommitted for brevity -->
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <RunConfiguration>
    <MaxCpuCount>0</MaxCpuCount>
    <BatchSize>1000</BatchSize>
    <ResultsDirectory>D:\a\_temp\TestResults</ResultsDirectory>
  </RunConfiguration>
</RunSettings>
**************** Starting test execution *********************

So it appears the VS test task is altering my runsettings on the fly and then not respecting the inclusions and exclusions values

My yaml is:

- task: VSTest@2
  displayName: 'Run Tests'
  inputs:
    testAssemblyVer2: '**\*UnitTests*.dll'
    searchFolder: '.\Output'
    codeCoverageEnabled: true
    runSettingsFile: .\Builds\coverlet.runsettings

Can anyone suggest a setting or some other way to respect the inclusions and exclusions?

Please check the log to see if there's any clue that the settings file is overrided. By the way, can you share the real test command executed by your task? — Jeff, May 07 '21 at 10:09
@Jeff The log shows the line `Updated Run Settings:` in the original question — ste-fu, May 10 '21 at 11:53
@Dmitresky It appears the codeCoverageEnabled flag does some weird stuff, my final solution was to run it via dotnet test and pass the settings in on the command line. [I posted a full solution on this question](https://stackoverflow.com/questions/60651663/can-you-use-coverlet-to-get-code-coverage-data-in-a-net-framework-project/67740275#67740275) — ste-fu, Nov 22 '21 at 14:34

score 2 · Answer 1 · answered Nov 23 '21 at 08:56

It is counterintuitive, but in order to VS test task does not alter runsettings on the fly, need to remove "codeCoverageEnabled: true" and add "runSettingsFile: .\Builds\coverlet.runsettings":

- task: VSTest@2
  displayName: 'Run Tests'
  inputs:
    testAssemblyVer2: '**\*UnitTests*.dll'
    searchFolder: '.\Output'
    runSettingsFile: .\Builds\coverlet.runsettings

Code coverage still will be calculated because of runsettings file.

In order to exclude something, you can check https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022#sample-runsettings-file I copy pasted here in case the link is broken

<!--
About include/exclude lists:
Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
Each element in the list is a regular expression (ECMAScript syntax). See /visualstudio/ide/using-regular-expressions-in-visual-studio.
An item must first match at least one entry in the include list to be included.
Included items must then not match any entries in the exclude list to remain included.
-->

            <!-- Match assembly file paths: -->
            <ModulePaths>
              <Include>
                <ModulePath>.*\.dll$</ModulePath>
                <ModulePath>.*\.exe$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
              </Exclude>
            </ModulePaths>

            <!-- Match fully qualified names of functions: -->
            <!-- (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.)  -->
            <Functions>
              <Exclude>
                <Function>^Fabrikam\.UnitTest\..*</Function>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
              </Exclude>
            </Functions>

            <!-- Match attributes on any code element: -->
            <Attributes>
              <Exclude>
                <!-- Don't forget "Attribute" at the end of the name -->
                <Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>

            <!-- Match the path of the source files in which each method is defined: -->
            <Sources>
              <Exclude>
                <Source>.*\\atlmfc\\.*</Source>
                <Source>.*\\vctools\\.*</Source>
                <Source>.*\\public\\sdk\\.*</Source>
                <Source>.*\\microsoft sdks\\.*</Source>
                <Source>.*\\vc\\include\\.*</Source>
              </Exclude>
            </Sources>

            <!-- Match the company name property in the assembly: -->
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>

            <!-- Match the public key token of a signed assembly: -->
            <PublicKeyTokens>
              <!-- Exclude Visual Studio extensions: -->
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>

VSTest@2 Task is Updating / Overriding coverlet.runsettings and not respecting exclusions

1 Answers1