BenchmarkDotNet InProcessEmitToolchain Complete Sample

Question

I'm looking at BenchmarkDotNet and benchmarking in general for the first time ever. I appear to be unable to run benchmarks using the normal BenchmarkRunner because of antivirus restrictions on our work laptops so I'm trying to use InProcessEmitToolchain, as documented here. However, in those samples and the ones listed here I see no entry point for the application that will actually trigger the benchmarks and I've gotten nowhere useful reading through the documentation.

Can anyone point me at a complete sample of how to use the InProcessEmitToolchain and/or jobs that explains how to use jobs in conjunction with an application entry point to run the tests?

Andreas Warberg · Accepted Answer · 2022-10-15T10:23:04.783

I was facing the same problem with antivirus (Windows Defender) blocking BenchmarkDotNet. I was able to figure out how to change the toolchain setup, though I had to use InProcessNoEmitToolchain since InProcessEmitToolchain was also blocked.

The example below did not actually trigger the antivirus, but it shows how to define which toolchain to use:

[Program.cs]

using BenchmarkDotNet.Running;
using Benchmarks;

_ = BenchmarkRunner.Run<MaterializeTest>();

[MaterializeTest.cs]

using BenchmarkDotNet.Attributes;

namespace Benchmarks;

[Config(typeof(AntiVirusFriendlyConfig))]
[MemoryDiagnoser]
public class MaterializeTest
{
    IEnumerable<int> _sequence;

    [Params(10, 100, 1000, 10000)]
    public int _size;

    [GlobalSetup]
    public void Setup()
    {
        _sequence = Enumerable.Range(0, _size).Select(i => Random.Shared.Next());
    }

    [Benchmark]
    public IReadOnlyList<int> ToList() => _sequence.ToList();

    [Benchmark]
    public IReadOnlyList<int> ToArray() => _sequence.ToArray();
}

[AntiVirusFriendlyConfig.cs]

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Toolchains.InProcess.NoEmit;

namespace Benchmarks;

public class AntiVirusFriendlyConfig : ManualConfig
{
    public AntiVirusFriendlyConfig()
    {
        AddJob(Job.MediumRun
            .WithToolchain(InProcessNoEmitToolchain.Instance));
    }
}

[Benchmarks.csproj]

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.2" />
  </ItemGroup>
</Project>

I'm not the OP but this gave me the information I needed to solve the same problem. Thanks. — Niya, Mar 25 '23 at 18:07

score 6 · Answer 2 · answered Feb 16 '23 at 12:03

You can tweak the configuration with the toolchains you like:

var config = DefaultConfig.Instance
    .AddJob(Job
         .MediumRun
         .WithLaunchCount(1)
         .WithToolchain(InProcessEmitToolchain.Instance));

Or alternatively

var config = DefaultConfig.Instance
    .AddJob(Job
         .MediumRun
         .WithLaunchCount(1)
         .WithToolchain(InProcessNoEmitToolchain.Instance));

And pass it to the Run method (building on the program from Andreas Warberg)

_ = BenchmarkRunner.Run<MaterializeTest>(config);

score 2 · Answer 3 · answered Jan 23 '23 at 22:00

2

I ended up being unable to solve this even using the InProcesNoEmitToolchain - our AV software was too persistent. I solved it by getting our IT folks to give us a folder whitelist entry in the AV software where we could run benchmarking.

answered Jan 23 '23 at 22:00

Thomas Parikka

472
5
17

score 1 · Answer 4 · answered Jul 17 '23 at 04:22

I had an issue where the app components being benchmarked was simply too slow; to circumvent this issue (the error message was takes too long to run. Prefer to use out-of-process toolchains for long-running benchmarks), I used the guidance given by other answers above, and extended it to the following:

public class MediumConfig : ManualConfig
{
    public MediumConfig()
    {
        AddJob(
               Job.MediumRun
                  .WithToolchain(InProcessNoEmitToolchain.Instance)
                  .WithStrategy(BenchmarkDotNet.Engines.RunStrategy.Throughput)
              )
            .WithOptions(ConfigOptions.DisableOptimizationsValidator);
    }
}

The config is then applied as usual:

[Config(typeof(MediumConfig))]
public class MyBenchmarkClassThatComparesALongRunningSetOfThings
{
    // ...
}

BenchmarkDotNet InProcessEmitToolchain Complete Sample

4 Answers4