23

I am running the NUnit tests (project in .Net Framework 4.5), as part of azure devops build pipeline.

- task: VSTest@2
  inputs:
    testAssemblyVer2: 'tests/**/*.Tests.dll'
    pathtoCustomTestAdapters: '$(Build.SourcesDirectory)/packages'
    codeCoverageEnabled: true
  displayName: 'NUnit Testing'

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: JaCoCo
    summaryFileLocation: '$(Common.TestResultsDirectory)/**/*.xml'
  displayName: 'Publish Code Coverage'
  //        summaryFileLocation: '$(Common.TestResultsDirectory)/**/*.coverage'

But I am not able to see the coverage report, all I see the download link for coverage results...

code coverage download link

How can I convert the .coverage report to JaCoCo format? OR generate the report directly in JaCoCo format?

I have seen some solution for .Net Core (link), but none for .Net framework

harishr
  • 17,807
  • 9
  • 78
  • 125

3 Answers3

38

Update:

As per the release to Azure Devops for Sprint 150

When publishing code coverage reports, you no longer need to specify HTML files.

Therefore, the script in my illustration no longer needs to use the report generator tool directly to create the html report, and when publishing the coverage results, the directory containing those html reports doesn't need to be specified.

Edit:


The trick I've found for getting the coverage results from a .Net Framework project to show up on the Code Coverage tab is in the same line of thought to your linked article.

  1. Don't run tests with the VS Test Task in Azure
  2. Install the Report Generator and Coverlet tools directly
  3. Use dotnet-vstest command for running tests through Coverlet
  4. Publish reports generated with Report Generator and Cobertura format coverage results


Don't use the VS Test Task

Running this task will allow you to collect coverage with a simple checkbox, but you then surrender your opportunity to provide the content for the Code Coverage Tab

no VsTest task



Install tools directly

Use a Powershell task (or similar) to install the Coverlet and Report Generator tools directly. This allows you to use them on projects that are not .Net Core.

"install tools:"
&dotnet tool install dotnet-reportgenerator-globaltool --tool-path . --version 4.0.12
&dotnet tool install coverlet.console --tool-path . --version 1.4.1



Use dotnet vstest through coverlet

It's my understanding that dotnet test doesn't play nice with .Net Framework projects/assemblies. However, we can still use the dotnet command, which we know will be on the agent machine, but we need to use it as a mechanism to get to the vstest.console.exe.

The Coverlet tool, as mentioned in the article you linked, will output coverage results in Cobertura format if you tell it to do so.

&$coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"


Publish results

use publish code coverage task



Complete script sample

note: this script is pretty rough, so use it as a thought exercise for your individual situation.

"install tools:"
&dotnet tool install dotnet-reportgenerator-globaltool --tool-path . --version 4.0.12
&dotnet tool install coverlet.console --tool-path . --version 1.4.1

"`nmake reports dir:"
mkdir .\reports

"`nrun tests:"
$unitTestFile = gci -Recurse | ?{ $_.FullName -like "*bin\*UnitTestProject2.dll" }
Write-Host "`$unitTestFile value: $unitTestFile"

$coverlet = "$pwd\coverlet.exe"

"calling $coverlet for $($unitTestFile.FullName)"
&$coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"

"`ngenerate report(s)"
gci -Recurse | 
    ?{ $_.Name -eq "coverage.cobertura.xml" } | 
    %{ &"$pwd\reportgenerator.exe" "-reports:$($_.FullName)" "-targetdir:reports" "-reporttypes:HTMLInline;HTMLChart" }

If you're struggling to figure out the escaping of quotes and such with the Coverlet command, YOU ARE NOT ALONE. I used the echoargs commandlet from PSCX more times than I care to admit so I could see what was actually getting provided to the .exe calls I was making.



The Results!!

...because that's really what matters

enter image description here



enter image description here




Original Answer:


Because of the way the linked article you mentioned is installing and using the report generator global tool I would think you can still follow those guidelines for creating the HTML inline and chart report types.

I'm not sure what is meant or how it works when the article says,

The point is the reporttypes: Use HTMLInLine for enabling the output on the Azure DevOps page. Azure DevOps Coverage page show index.html on the web.

I'm understanding that you can use the tool to create the HTML report from the .xml coverage results, and then publish the coverage results and report together with the Publish Code Coverage task.

So it seems all you need is to have an .xml format of the .coverage tool.

I didn't get it working in straight powershell, but you could follow the instructions from the Report Generator documentation to write a C# utility to access the Coverage.Analysis library.

Community
  • 1
  • 1
Josh Gust
  • 4,102
  • 25
  • 41
  • 2
    Coverlet documentation mentions the `--no-build` option given to `dotnet test`. This is not an option for `dotnet vstest` and including it will produce failing output. The `dotnet vstest` chain will expect the assemblies to be build when the tool is used, and therefore has no risk of rebuilding the assembly after **Coverlet** has added the coverage instrumentation. – Josh Gust Feb 21 '19 at 17:10
  • 1
    Don't forget the **merge** feature in coverlet if you have multiple coverage results, because having multiple `index.html` files generated by report generator wouldn't work for showing the files on the coverage tab. **Also**, the `--include` and `--exclude` options for coverlet are really nice to use. – Josh Gust Feb 22 '19 at 14:43
  • @JoshGust thanks for your explanation. Everything works fine but in the end I don't get a `Code Coverage` tab. I have to open the reports manually from the agent directory. Files wich get generated are: `index.html`, `index.htm`, `Class1.htm`, `Class1_Test.htm` and `CoverageHistory.htm`. I have only one `Class.cs` and a corresponding `Class_Test.cs`. The content of the `index.html` also looks empty. Any idea what went wrong? – Mar Tin Jul 02 '19 at 11:58
  • This answer is so expansive that I wish I could upvote more than once. Thank you @JoshGust – Martin Noreke Jul 02 '19 at 15:05
  • @MarTin because you say `I have to open the reports manually from the agent directory` I'm assuming you're using Azure DevOps **Server** (2019?) as opposed to Azure DevOps **Services**, correct? In that case, perhaps the following community thread is the reason you are [not seeing the coverage tab](https://developercommunity.visualstudio.com/content/problem/578111/code-coverage-tab-missing-in-azure-devops-server.html). As for the seemingly empty reports, troubleshoot to confirm the data in the .xml output, but with only 2 classes I wouldn't expect a lot of data. – Josh Gust Jul 02 '19 at 22:44
  • Thank you for the very detailed answer! I'm just stuck on coverlet. When running this on my local machine it runs the tests and generates a cobertura xml file just fine. When running this command on the build machine it runs the tests correctly but the outputed cobertura file is empty. Any ideas? – ds99jove Aug 19 '19 at 08:19
  • @JoshGust - thx for sharing, this works great, but I lost my test results in the Test tab. So I added in the **Publish Test Results** task with `**\TestResults\*.trx` before the publish code coverage task and they came back – Stinky Towel Aug 26 '19 at 19:06
  • @StinkyTowel - I think I made this change to my pipeline and haven't made the change to the answer. Also, stay tuned for an extension that wraps all this up in a neat little package. – Josh Gust Aug 26 '19 at 20:38
  • @JoshGust I did follow your steps and I get this error in the shell script "Program 'coverlet.exe' failed to run: The filename or extension is too longAt " Any help would be appreciated. – Chameera Dulanga Nov 18 '19 at 16:03
  • @JoshGust thank you sir, your script solution worked flawlessly. – sanjeev Apr 12 '20 at 15:27
  • @JoshGust I have a bit of a funny issue. The results are generated, they show up in artifacts but the code coverage tab only gives me a download link. When I open the index.html file, it renders nicely, but Azure DevOps opted not to use them for some weird reason. I have only one test project so there are no conflicts or need for merging. Any pointers are appreciated. – Ε Г И І И О May 26 '21 at 11:13
2

For anyone looking for code coverage in Azure Devops (using classic editor, without Yaml), in current .NET (core) 5, with xUnit tests:

  1. In your xUnit test project, add following (it generally comes by default in .NET 5, xUnit template now):

    <PackageReference Include="coverlet.collector" Version="3.0.3" />

    Keep checking for new version.

  2. Head to Azure devops, create pipeline using classic editor. Do the restore, build steps. (Or you can choose dotnet core template as below): select dotnet core pipeline template

  3. In the test command of dotnet core task, add argument - --collect:"XPlat Code Coverage". Remember "XPlat Code Coverage" is friendly name and case sensitive. Your test command would look like: enter image description here Check this checkbox if you want to publish your test results: Publish test results and code coverage, but it won't publish code coverage. The functionality is not yet working (at least not in non-windows).

  4. Next add - Publish code coverage results task. Choose "Code coverage tool" as "Cobertura" and in "Summary file" field, add $(Agent.TempDirectory)/**/coverage.cobertura.xml. Looks like this: dotnet test task config in pipeline

  5. Save and Queue (in any agent, I use Ubuntu) and see the result once pipeline run completes: pipeline run result

  6. Coverage report tab: code coverage summary

  7. HTML Coverage reports and coverage cobertura xml are published as artifacts: coverage reports

Meer
  • 678
  • 6
  • 12
  • But do you see an HTML report when you navigate to the Code Coverage tab? – Ε Г И І И О Aug 21 '21 at 05:37
  • 1
    Yes, the HTML coverage reports and coverage cobertura xml are published as artifacts and the coverage summary is embedded in Code Coverage tab. Updated the posts with screenshots. – Meer Aug 21 '21 at 11:14
0

You can use Publish Code Coverage Results task in azure devops pipeline to see code coverage result in Jacoco format.

for further information about setup and configuration , please check the blog in MSDN

https://learn.microsoft.com/hi-in/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=tfs-2015#q--a

Hope it helps.

Mohit Verma
  • 5,140
  • 2
  • 12
  • 27
  • Please look at my sample code, i am already using it but its still not working – harishr Feb 21 '19 at 11:30
  • This doesn't work unless one has a way to get the code coverage results from netFW test runs into Cobertura ro Jacoco format. – Josh Gust Mar 19 '19 at 15:26