1

Currently I am trying to get results from SpecFlow Acceptance Tests and integrate them with Pickles Docs. What I have is, on build generated pickles site. What happens then is on each test I have this icon with text "Inconclusive":

enter image description here

and then if I go to scenario summary by result I have result:

enter image description here

So in fact it's showing all scenarios, but it's not running them to see which are passing and which failing.

Right now I get to PicklesDocs documentation, which I think will do what I need, it's here. Still when I execute command Pickles.exe --test-results-format=xunit or Pickles.exe -trfmt=xunit what I receive is: enter image description here

Still after that it's generating file "index.html", but it's only showing tests without any result.

enter image description here

Do you know how I can configure this to work? Should I use maybe NUnit test result or generate results with specflow.exe nunitexecutionreport, or maybe pickles docs can do this as well?

Note: I am implementing steps using NUnit.

Thanks

Johana
  • 45
  • 1
  • 5

3 Answers3

2

I'm the maintainer of Pickles and I'll try to get you on your way.

First off, Pickles doesn't run the tests for you. You will need to run the tests first, and make a note of the location of the output file of the tests.

Once you have the test result file, you need to instruct Pickles to use it by adding the --link-results-file parameter (or --lr).

You will also need to tell Pickles which unit test framework and version you used when running the result file. You write that you're using NUnit. Assuming you are using the latest version of NUnit (or at least a 3.x version), the correct value for the --trfmt parameter is nunit3.

Does that help? Please ask further questions if you need more help.

MMehta
  • 45
  • 4
Dirk Rombauts
  • 136
  • 4
  • Hello and thank you for your response, this help me quite a lot, but the next thing is, would it work with ".trx" file or converted to xml? Because I have the following previous issue https://stackoverflow.com/questions/46051922/c-sharp-nunit-console-test-generation. So after all I would summarize as, I got .NET Core 2.0 assembly with acceptance tests - SpecFlow + NUnit(could be replaced with other framework) and I want to receive result files, so I can integrate or use it inside pickles site. So having success/fail tests online. Thanks – Johana Sep 05 '17 at 12:44
  • Pickles supports these test result formats: NUnit 2.x, NUnit 3.x, xUnit 1.x, xUnit 2.x, MSTest, VSTest (MSTest and VSTest both use .trx files but they are different on the inside!) Cucumber (both Ruby and JS), and SpecFlow+ Runner (formerly known as SpecRun). I have no experience with a .NET Core setup, though. – Dirk Rombauts Sep 05 '17 at 12:57
  • Executing command "pickles.exe --feature-directory=.\'Acceptance Tests' --output-directory=.\docs --link-results-file=TestResults\TestResult.trx --test-results-format=nunit3 --documentation-format=html" seems to work. So using TestResult.trx it doesn't throw exception, but in the result index.html I still have only scenarios, but without any report saying if they're successful or they are failing. – Johana Sep 05 '17 at 13:03
  • Now I tried "pickles.exe --documentation-format=dhtml --feature-directory= --output-directory=.\documentation --link-results-file="\TestResults\TestResult.xml" --test-results-format=nunit3", so don't generate results as ".trx", but as ".xml" and when I open "index.html" generated file and here's again what I receive => "http://prntscr.com/ghdn0o" and "http://prntscr.com/ghdn8t" – Johana Sep 05 '17 at 13:47
  • This is the output from the console => "http://prntscr.com/ghdnu5". The important thing here I think is that I have "Incorporate Test Results?": "Yes", where in previous command line executions, that was "No". – Johana Sep 05 '17 at 13:49
  • It's important that the format of the test results file (TestResults.trx or TestResults.xml) matches the value of the --test-results-format parameter. Pickles cannot guess the format. Pickles will only show test results if the value is correct. Which unit test framework did you use to create the TestResult.xml file? – Dirk Rombauts Sep 06 '17 at 07:31
  • I am using NUnit – Johana Sep 07 '17 at 06:15
  • So still when I generate pickles-site using ".trx" or ".xml" I have "Incorporate Test Results?": "Yes" => http://prntscr.com/gi3l4z – Johana Sep 07 '17 at 06:26
  • As well here is generated TestResults.xml file => http://prntscr.com/gi3nr2 using "dotnet test --logger "trx;LogFileName=TestResult.xml"" command. – Johana Sep 07 '17 at 06:34
  • What I also did is replacing NUnit with MSTest framework, and on PIckles generation I am setting "--test-results-format=mstest", also tried with "--test-results-format=vstest", still no result. – Johana Sep 07 '17 at 07:23
  • Once try with ".trx" and one with ".xml" => http://prntscr.com/gi4a7m => http://prntscr.com/gi4aeg – Johana Sep 07 '17 at 07:35
  • So you are using "dotnet test" to produce the TestResult file, am I right? Unfortunately, Pickles doesn't support dotnet test (yet). – Dirk Rombauts Sep 07 '17 at 07:54
  • Alright, now I understand. Do you think there is any other option to generate TestResult with NUnit or any other framework on .NET Core library? – Johana Sep 07 '17 at 08:10
  • No, there's currently no .NET Core compatibility for Pickles. Pickles is open source, so feel free to make a contribution that adds this functionality. That's usually the fastest way to make it happen :-) – Dirk Rombauts Sep 07 '17 at 08:20
  • I am considering making contribution for .NET Core. And just last question, does it support .NET Standard? – Johana Sep 07 '17 at 08:26
  • No, no support for .NET Standard yet either. If you want to contribute, please have a look at https://github.com/picklesdoc/pickles/blob/develop/CONTRIBUTING.md, especially the section about contributing to the test result providers. – Dirk Rombauts Sep 07 '17 at 08:28
  • Thank you, will check it out. – Johana Sep 07 '17 at 08:32
0

Currently Pickles Docs doesn't support test results output coming from assembly targeting .NET Core or .NET Standard. Could be made contribution on that https://github.com/picklesdoc/pickles/blob/develop/CONTRIBUTING.md

Johana
  • 45
  • 1
  • 5
0

The only solution I found to use Pickles with .Net Core, is to use another logger such as XunitXml.TestLogger here. You will have to:

  • Reference this Nuget package
  • run your tests: dotnet test --logger:"xunit;LogFilePath=TestsResults.xml"
  • run pickles using format xunit2
Loul G.
  • 997
  • 13
  • 27