12

I'm benchmarking some .net framework stuffs, I'm using .net framework, C# and BenchmarkDotNet

What I want to do is; I'm writing a lot of benchmark tests and I'm only interested in summary sections of the reports. How can I configure BenchmarkDotNet to show only summary section of tests?

Here is a screenshot to be more clear;

enter image description here

AndreyAkinshin
  • 18,603
  • 29
  • 96
  • 155
Lost_In_Library
  • 3,265
  • 6
  • 38
  • 70

2 Answers2

4

Why do you want skip logs? Benchmarks could take a lot of time, if you disable logs, you will look at the black screen for some time. If something goes wrong, then you will not know about it.

However, there is a workaround. BenchmarkDotNet uses special Configs for customization. Loggers are a part of these Configs. If you don't specify any config, the default one will be using it. You can easily extend it, but there is no nice API to disable a part of the default config (hopefully, will be added soon; the corresponded API is in the discussion stage right now). So, you have to define own config, add all parts of the default config except loggers and pass it to BenchmarkRunner. Then the ConsoleLogger will not be used. After that, you have to print the summary table and conclusions to console manually. Also, the full log and the summary table in markdown format will be in the BenchmarkDotNet.Artifacts folder.

The source code:

var config = new ManualConfig();
config.Add(DefaultConfig.Instance.GetColumnProviders().ToArray());
config.Add(DefaultConfig.Instance.GetExporters().ToArray());
config.Add(DefaultConfig.Instance.GetDiagnosers().ToArray());
config.Add(DefaultConfig.Instance.GetAnalysers().ToArray());
config.Add(DefaultConfig.Instance.GetJobs().ToArray());
config.Add(DefaultConfig.Instance.GetValidators().ToArray());
config.UnionRule = ConfigUnionRule.AlwaysUseGlobal; // Overriding the default

var summary = BenchmarkRunner.Run<TestBench>(config);

var logger = ConsoleLogger.Default;
MarkdownExporter.Console.ExportToLog(summary, logger);
ConclusionHelper.Print(logger, config.GetCompositeAnalyser().Analyse(summary).ToList());
AndreyAkinshin
  • 18,603
  • 29
  • 96
  • 155
  • 2
    Hi @AndreyAkinshin Thanks for your answer and thanks for your awesome benchmark tool. I want to skip detailed logs because console application window is text size limited. So if the developer writes a lot of benchmark tests, he/she should leave the vs and take look at the exported summary files. I don't want to do that while I'm only interested summary results quickly. Because I'm changing code constantlly. Otherwise, detailed logs are awesome. Maybe a feature like this will be helpfull; an attribute perheps; "[DetailLevel.SummaryOnly]" or "[DetailLevel.SummaryAndSpec]" or "[DetailLevel.Full]" – Lost_In_Library Feb 28 '17 at 08:49
  • I'm glad that the library is useful for you. Consider extending the text size limit of your console window. I will add to nice API for disabling ConsoleLogger and will improve printing of the summary table. Don't think that it's good idea to introduce an additional attribute for log details because it's an uncommon use case. – AndreyAkinshin Feb 28 '17 at 13:35
  • 1
    Just to mention I have been on a hunt to try and do same thing, I just want to show every summary table (not joined as one) in a list and every avenue seems to lead me to something that no longer functions the way the examples provided show. It would be great if in the docs you could show a simple example of running multiple tests and just dumping the summaries out even if its just in the console to allow quicker iteration and quick feedback for those scenarios where you dont care about the artifacts, you just want to change something and see if it is a step in the right direction. – Grofit Apr 30 '22 at 20:50
3

AndreyAkinshin answeser is really great, but one part isnt valid anymore! ManualConfig doesnt provide .GetCompositeAnalyser() anymore, so you have to get the ImmutableConfig and thats only possible to form the BenchmarksCase

For example if you are only running one Benchmarkcase: you can do this with ...First()

this fixed it for me

ConclusionHelper.Print(logger, summary.BenchmarksCases.**First()**.Config.GetCompositeAnalyser().Analyse(summary).ToList());
Crimson
  • 103
  • 6
  • Thanks for this. Unfortunately, `GetColumnProviders`, etc are now obsolete and we are supposed to use `AddColumnProvider`, etc. However, a simple replacement does not work. Any ideas? Much appreciated. –  Apr 19 '21 at 08:11