No code coverage in VSTS when .runsettings file supplied

Question

We're having trouble with code coverage on our Visual Studio Online (VSTS) build definition, where no results are returned when we define a custom .runsettings file.

Locally this is working fine, however, when we update our build definition on Visual Studio Online with this file, it doesn't return any results.

My .runsettings file, I left everything in there per an example I found.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
    <TargetPlatform>x86</TargetPlatform>
    <TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
    <TreatTestAdapterErrorsAsWarnings>false</TreatTestAdapterErrorsAsWarnings>
    <CollectSourceInformation>true</CollectSourceInformation>
    <MaxCpuCount>0</MaxCpuCount>
  </RunConfiguration>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Include>
                <ModulePath>.*\.dll$</ModulePath>
                <ModulePath>.*\.exe$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                <ModulePath>.*TestAdapter.*</ModulePath>
                <ModulePath>.*xamarin.*</ModulePath>
              </Exclude>
            </ModulePaths>
            <Functions>
              <Exclude>
                <Function>^Fabrikam\.UnitTest\..*</Function>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
              </Exclude>
            </Functions>
            <Attributes>
              <Exclude>
                <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
                <Attribute>^NUnit.Framework.TestFixtureAttribute$</Attribute>
                <Attribute>^Xunit.FactAttribute$</Attribute>
                <Attribute>^Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute$</Attribute>
              </Exclude>
            </Attributes>
            <Sources>
              <Exclude>
                <Source>.*\\atlmfc\\.*</Source>
                <Source>.*\\vctools\\.*</Source>
                <Source>.*\\public\\sdk\\.*</Source>
                <Source>.*\\microsoft sdks\\.*</Source>
                <Source>.*\\vc\\include\\.*</Source>
              </Exclude>
            </Sources>
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>
            <PublicKeyTokens>
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <CaptureTraceOutput>false</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
  </MSTest>
</RunSettings>

Update 1

Here is the output of the same run settings file on my local environment, attached to Visual Studio as the file to use:

and

What this settings file does is, to exclude the Xamarin.Forms.*.dlls from the analysis, as we should not be covering these assemblies in our project.

As you can see, it works locally, but when I use this settings file in build run in VSTS, it gives me the following:

I have tried skimming through the logs, and cannot see anything wrong with what we've done. I am in the process of setting up an on-premise TFS 2017 installation to test the same, at least I'll have access to the output directories to see what happens during the build.

I'm not familiar with this, so I'm taking stabs at any possible answer.

Do you specify the runsetting when do test locally? (Test=> Test Setting) Can you share the detail log on the OneDrive? (Set system.debug to true and queue build) — starian chen-MSFT, Nov 02 '17 at 03:04
@starain-MSFT I did apply it locally, and it works. I get a successful code coverage analysis report, but the same run settings file does not work on VSTS. I've updated my original question with the update — JadedEric, Nov 02 '17 at 07:07
I can. Just the VS Test run log should be sufficient, right? — JadedEric, Nov 02 '17 at 07:45
Yes, just VS Test log. (set system.debug to true, then queue build) — starian chen-MSFT, Nov 02 '17 at 07:46
@starain-MSFT the log file here: https://1drv.ms/f/s!AtTQ9jXAPOTxgbBzWyQEQZezL8KTJw — JadedEric, Nov 02 '17 at 07:49
Based on the log, there is `.*Tests.dll` in Exclude section, try to remove it and try again. — starian chen-MSFT, Nov 02 '17 at 08:29
@starain-MSFT it now has results, but it's testing only two Xamarin assemblies. My project files are completely ignored, which is way we decided to invest time in runsettings. It's just not working for us. I have two projects in the solution for which I need coverage TestApp and TestApp.Models, additionally, I have TestApp.UnitTests which is the test project. The latest test run, with the *Tests.dll removed, it only covers Xamarin.Forms.Core.dll and Xamarin.Forms.Xaml.dll which is not what I need coverage for. This is so confusing :( — JadedEric, Nov 02 '17 at 08:45
What's the result if you run the test with CodeCoverage enabled through vstest.console.exe tool manually? — starian chen-MSFT, Nov 02 '17 at 09:39
Can you reproduce this issue with the new projects? You can share the sample projects on the OneDrive. — starian chen-MSFT, Nov 02 '17 at 09:41
Do you have .Net Core or .Net Standard projects? Than you have to go to `project properties > Build Tab > "Advanced..." button > Debugging information Full or at least Pdb-only`. I have got the same problem and searched nearly 4h until I found a solution. Hope it will solve your issue too. — ChW, Mar 20 '18 at 16:41
Did you find a solution to this problem? We're currently experiencing the same issue with VSTS and supplying "Other console options" with /Settings:. — Skintkingle, Jul 23 '18 at 13:32
@Skintkingle unfortunately not. We gave up on this, and went the Sonarcloud route. — JadedEric, Aug 20 '18 at 11:18

Kraego · Answer 1 · 2021-09-13T09:08:02.960

I have the following Solution on AzureDevOps OnPremise 2020, with an asp .net app. It works fine in mine scenario, where I have to transform the results to Cobertura (for sonarqube). I think for you the important task is the 'dotnet test' task (see below):

Here are the build pipeline parts:

 - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test
      arguments: '--configuration $(buildConfiguration) --no-build --logger trx --collect "XPlat Code coverage" --settings coverlet.runsettings'
      projects: '**/*Test/*.csproj'
      publishTestResults: true
  
  - task: DotNetCoreCLI@2
    displayName: Install ReportGenerator tool
    inputs:
      command: custom
      custom: tool
      arguments: update --tool-path . dotnet-reportgenerator-globaltool --ignore-failed-sources
  
  - script: reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.opencover.xml -targetdir:$(Build.SourcesDirectory)/coverlet/reports -reporttypes:"Cobertura"
    displayName: Transform coverage reports
  
  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: '$(Build.SourcesDirectory)/coverlet/reports/Cobertura.xml'
      failIfCoverageEmpty: false

Coverate on azureDevOps:

No code coverage in VSTS when .runsettings file supplied

1 Answers1