1

Environment: Visual Studio 2013 Premium, Win7Ultimate, CodeCoverage.exe

Goal: Code Coverage Report that excludes test project code to later be converted to a .coveragexml for reporting to SonarQube 5.1.

Annoyance I wouldn't even know of this parse error without adding the /verbose switch to the command. My only indication of a failure was the .coverage file was no longer being generated when I added the /config switch.

File Works in VS2013 IDE: MyProject.runsettings file provides the expected output using "Analyze Code Coverage" in the IDE.

Menu: Test | Test Settings | Select Test Settings File... MyProject.runsettings

Menu: Test | Analyze Code Coverage | All Tests

Attempting to run the CodeCoverage.exe file to generate code coverage for my tests I can't seem to use ANY *.runsettings files without getting an error: 

"Error: Failed to parse configuration file <configfile>.runsettings"

Path Definitions:

codeCoveragePath = C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Dynamic Code Coverage Tools

vstestpath = C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow

myProjectOutputPath = assume correct since I get results when not using /config switch

Run Command receiving Error (assume paths are correct): Note: I'm not showing with /verbose switch since I shouldn't be using it under working conditions

%codeCoveragePath%\CodeCoverage.exe collect /config:MyProject.runsettings /output:CoverageOutput.coverage %vstestpath%\vstest.console.exe %myProjectOutputPath%\MyClass.Tests.Unit.dll

Exe Works if I DON'T use the /config option If I remove the /config:MyProject.runsettings from the run command, I get a full report that includes the test project, but that let's me know the rest of the command is correct, it just doesn't like the runsettings file.

I've tried using the following examples:

Visual Studio 2013 runsettings Template file WITHOUT modification

MSDN's sample file

Completed blank file, no content: error

File with only the xml declaration: error

File with only RunSettings Node declared: error

I've even used the Troubleshooting tips from MSDN, too: no help.

MyProject.runsettings file: 

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>.*\.Tests\.Unit\.dll$</ModulePath>
              </Exclude>
            </ModulePaths>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

The file seems to be correct based on the fact that the IDE will use it and generate the correct output in the "Code Coverage Results" window by only reporting the MyClass code and not any MyClass.Tests.Unit code.

I'm at the point that I think it is the CodeCoverage.exe command line doesn't like the /config option or it is using a different xml schema.

Update

Works gives the output I want, just can't specify file location for next step

%vstestpath%\vstest.console.exe /Settings:MySettings.runsettings %myProjectOutputPath%\MyClass.Tests.Unit.dll

Doesn't Work Gives exact opposite output I want (only test.dll coverage in the report).

%codeCoveragePath%\CodeCoverage.exe collect /output:CoverageOutput.coverage %vstestpath%\vstest.console.exe /Settings:MySettings.runsettings %myProjectOutputPath%\MyClass.Tests.Unit.dll

Still looking for an answer.

Dan M.
  • 29
  • 1
  • 6
  • FYI, the SonarQube C# 4.1 plugin now correctly ignores code coverage on test projects - even if it is present in the report, see http://jira.sonarsource.com/browse/SONARNTEST-16 - this might simply your life :-) – Dinesh Bolkensteyn Aug 04 '15 at 14:03
  • I've been upgrading to SonarQube 5.2 and it has C# 4.3 installed and it is ignoring all the test projects as I would like to see. However, I'm still dealing with the original problem. The command line described is ONLY generated code coverage reports for the test projects and a C++ project (the larger actual project, not this simplified one). The IDE using the same exact.runsettings file reports exactly as I would expect. The CodeCoverage.exe analyze command then converts it to a usefule output as well that SonarQube is able to consume. – Dan M. Jan 26 '16 at 16:39
  • UPDATE: I manually used the vstest command without the codecoverage.exe collect command using the /Settings:mytestsettings.runsettings file and I'm getting the output I would like albeit I have no idea where that file will end up. The analyze command gives a good output (no test projects) However, the problem still stems around the ability to pass in the runsettings file to the codecoverage.exe collect command line and have it generate a correct .coverage output. As of right now, I'm only generating .coverage for exactly opposite of what I want resulting in a completely useless file. – Dan M. Jan 26 '16 at 16:43
  • For clarity: 'code' %CodeCoverageExePath% collect %VsTestExePath% /Settings:mySettings.runsettings /EnableCodeCoverage /logger:trx %listOfTestAssemblies 'code' %CodeCoverageExePath% collect %VsTestExePath% /Settings:mySettings.runsettings /EnableCodeCoverage /logger:trx %listOfTestAssemblies – Dan M. Jan 26 '16 at 19:32

3 Answers3

2

I believe you need to specify the runsettings file at the back of the vstest.console.exe using the /Settings: flag (as opposed to the config flag of the CodeCoverage.exe).

So that your command becomes:

%codeCoveragePath%\CodeCoverage.exe collect /output:CoverageOutput.coverage %vstestpath%\vstest.console.exe %myProjectOutputPath%\MyClass.Tests.Unit.dll /Settings:MyProject.runsettings

Michael Freidgeim
  • 26,542
  • 16
  • 152
  • 170
Reddog
  • 15,219
  • 3
  • 51
  • 63
  • I had also tried that but thought my question was getting a bit long in the tooth. When I did that, I got no output, plus that would be moving the /config: switch to the run of the vstest.console.exe instead of the codecoverage.exe. Thank you for the thought. – Dan M. Apr 10 '15 at 15:46
0

I was having the same problem and found your question when searching for some info on my error. My assumption had been also that the config file format was the same as the .runsettings used by vstest.console.exe but from the parse error after adding /verbose I then suspected it was a different format so had a look to see if there was a default config for CodeCoverage.exe to see what it looked like, and I did find one at:

C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.config

And the format appears to just be the inner <CodeCoverage> part of the .runsettings format

I have now got filtering to work but needed to copy over all the filter items from the default config as they now don't get loaded, so I ended up using a config like the following:

<CodeCoverage>

  <ModulePaths>
    <Exclude>
    <ModulePath>.*\\unittests.dll</ModulePath>
    </Exclude>
  </ModulePaths>

  <Sources>
    <Exclude>
      <!--extracted from default CodeCoverage.config -->
      <Source>.*\\atlmfc\\.*</Source>
      <Source>.*\\vctools\\.*</Source>
      <Source>.*\\public\\sdk\\.*</Source>
      <Source>.*\\externalapis\\.*</Source>
      <Source>.*\\microsoft sdks\\.*</Source>
      <Source>.*\\vc\\include\\.*</Source>
    </Exclude>
  </Sources>

  <Functions>
    <Exclude>
      <!--extracted from default CodeCoverage.config -->
      <Function>^std::.*</Function>
      <Function>^ATL::.*</Function>
      <Function>.*::__GetTestMethodInfo.*</Function>
      <Function>.*__CxxPureMSILEntry.*</Function>
      <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
      <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
      <Function>.*::YOU_CAN_ONLY_DESIGNATE_ONE_.*</Function>
    </Exclude>
  </Functions>

</CodeCoverage>

and command line:

CodeCoverage collect /output:coverage.dat /config:coverage.settings vstest.console unitTests.dll /Logger:trx /Settings:test.runsettings
Glen
  • 459
  • 7
  • 6
  • wow... if this actually works then that is yet another maintenance item to have to manage and keep in sync. yuck. Unfortunately, I have long abandoned this effort finding other ways to accomplish my task. Not really wanting to mark this as an answer since I can't confirm. – Dan M. Jul 20 '15 at 13:30
  • See updated description: bottom line, this doesn't work. – Dan M. Jan 26 '16 at 19:55
0

Okay, Here comes the HACK!!!!

Basic steps:

  • Find and delete all *.coverage files
  • Run vstest command WITHOUT [codecoverage.exe collect] wrapper
  • Find the new *.coverage files and send to [codecoverage.exe analyze] command

Details

I updated the build.proj file I've been using to execute all this to do the Basic steps: 

<PropertyGroup>
  <SqCodeCoverageResultsFile>VisualStudio.coveragexml</SqCodeCoverageResultsFile>
</PropertyGroup>
<Target Name="BuildTestAssemblyList"
      BeforeTargets="RunAllTestsWithCodeCoverageAndConvertToXmlOutput">
  <CreateItem Include="**\*.Tests.Unit.dll">
    <Output TaskParameter="Include" ItemName="TestAssemblies" />
  </CreateItem>
</Target>
<Target Name="BuildCoverageFileList"
      BeforeTargets="RunAllTestsWithCodeCoverageAndConvertToXmlOutput">
  <CreateItem Include="**\*.coverage">
    <Output TaskParameter="Include" ItemName="CoverageFiles" />
  </CreateItem>    
</Target>
<Target Name="RunAllTestsWithCodeCoverageAndConvertToXmlOutput">
  <Delete Condition="Exists($(SqCodeCoverageResultsFile))" Files="$(SqCodeCoverageResultsFile)" />
  <Delete Files="@(CoverageFiles)" />
  <Exec Command="&quot;$(VsTestExecutable)&quot; /EnableCodeCoverage /Settings:MyProject.runsettings /inIsolation /logger:trx  @(TestAssemblies->'&quot;%(FullPath)&quot;',' ') " /> 
  <CreateItem Include="**\*.coverage">
    <Output TaskParameter="Include" ItemName="NewCoverageFiles" />
  </CreateItem>    
  <Exec Command="&quot;$(VsCodeCoverageExecutable)&quot; analyze /output:&quot;$(SqCodeCoverageResultsFile)&quot; @(NewCoverageFiles->'&quot;%(FullPath)&quot;',' ') " />
</Target>`

Now running the CodeCoverage.exe analyze command with those found *.coverage files will now output to the same filename I was trying to achieve before and get the results I wanted.

MSBuild.SonarQube.Runner.exe gets what it wants, I have the results I want, and the world can start to revolve again =)

Improvement: I could use a CustomTask and search for the perfect or most recent or whatever logic you can think of for that correct single file so I wouldn't have to delete all my other *.coverage files. I could, but I didn't because this is supposed to be run on a build server that shouldn't have that kind of history laying around anyways in my opinion.

Dan M.
  • 29
  • 1
  • 6