5

I'm trying to run benchmarks created with BenchmarkDotNet within Azure DevOps pipeline. Here is a build where I have problems. Take a look at Run benchmarks step. As you can see all tests with benchmarks are passed but actually none benchmark executed. Message from log:

// BeforeAnythingElse
ExitCode != 0

If I add InProcess attribute to classes with benchmarks then benchmarks will be executed. But some of my benchmarks are long running so they are interrupted by BenchmarkDotNet with recommendation to use out-of-process benchmarks.

How can I run out-of-process benchmarks within Azure pipelines?

Maxim
  • 1,995
  • 1
  • 19
  • 24
  • I saw the latest buid run was successful. Did you find a solution to this issue? – Levi Lu-MSFT Aug 12 '19 at 08:58
  • No, I use `InProcess` attribute but it has limitations like max time for benchmark. Right now I'm OK with in-process benchmarks, but the problem with out-of-process ones still exist. I'd like to use out-of-process benchmarks but seems they don't work on Azure DevOps build agents. – Maxim Aug 12 '19 at 11:09
  • When you run out of process benchmarks locally on VisualStudio,did you get the same error with azure devops? I tried locally and on azure devops but failed to reproduce the error above. Can you provide a new link to the error build log? as the origin link you provided is obsolete. – Levi Lu-MSFT Aug 19 '19 at 10:12
  • Locally out-of-process benchmarks work fine. I'll create out-of-process benchmark a bit later and provide a link to build log. – Maxim Aug 19 '19 at 10:17
  • Here fresh build: https://dev.azure.com/mhallmuz/DryWetMIDI/_build/results?buildId=88. Step **Run benchmarks**, test method **Run** (which executes benchmark). – Maxim Aug 20 '19 at 10:41
  • sorry for late reply. I ran the outofprocess benchmark tests on local visual studio. i got the same result with the vstest run result log from above build you provided. I also run these benchmark tests on my azure devops pipeline and got the same result with locally run. the time that these bench tests run took is about the same too. I aslo got `Exception NUnit.Engine.NUnitEngineException`, this exception happened is because the benchtest.dll in obj folder was included in the vstest task.after the obj folder is excluded(`!**\obj\**`) , the benchmark tests ran without any exception. – Levi Lu-MSFT Aug 29 '19 at 09:03
  • I wonder how i can tell if a benchmark test is actually executed or not? – Levi Lu-MSFT Aug 29 '19 at 09:09
  • I suppose we can pull this info from benchmark result and throw assert exception to manually fail a test. – Maxim Aug 29 '19 at 09:39
  • I am sorry, From trx test results file which generated by Vstest task, i can see the benchmarks ran. But it only showed the passed message in the console,not information in trx file as describe in this [thread]( https://stackoverflow.com/questions/44767110/vstest-console-when-run-with-loggerconsole-does-not-print-debug-output-from-te). Is it the case that you indicated? – Levi Lu-MSFT Sep 03 '19 at 10:23

0 Answers0