Is it possible to use #if NET6_0_OR_GREATER to exclude a benchmark method from a BenchmarkDotNet run?

Question

Suppose that you're writing some benchmarks for use with BenchmarkDotNet that are multi-targeted to net48 and net6.0, and that one of those benchmarks can only be compiled for the net6.0 target.

The obvious thing to do is to use something like this to exclude that particular benchmark from the net48 build:

#if NET6_0_OR_GREATER

[Benchmark]
public void UsingSpan()
{
    using var stream = new MemoryStream();
    writeUsingSpan(stream, _array);
}

static void writeUsingSpan(Stream output, double[] array)
{
    var span  = array.AsSpan();
    var bytes = MemoryMarshal.AsBytes(span);

    output.Write(bytes);
}

#endif // NET6_0_OR_GREATER

This unfortunately doesn't work, and the way in which it doesn't work depends on the order of the targets specified in the TargetFrameworks property in the project file.

If you order the frameworks so that net6.0 is first as <TargetFrameworks>net6.0;net48</TargetFrameworks> then (in the example above) the UsingSpan() method is included in BOTH targets, resulting in BenchmarkDotNet build errors for the net48 target and output such as this:

|            Method |                Job |            Runtime |       Mean |     Error |    StdDev |
|------------------ |------------------- |------------------- |-----------:|----------:|----------:|
| UsingBitConverter |           .NET 6.0 |           .NET 6.0 | 325.587 us | 2.0160 us | 1.8858 us |
|      UsingMarshal |           .NET 6.0 |           .NET 6.0 | 505.784 us | 4.3719 us | 4.0894 us |
|         UsingSpan |           .NET 6.0 |           .NET 6.0 |   4.942 us | 0.0543 us | 0.0482 us |
| UsingBitConverter | .NET Framework 4.8 | .NET Framework 4.8 |         NA |        NA |        NA |
|      UsingMarshal | .NET Framework 4.8 | .NET Framework 4.8 |         NA |        NA |        NA |
|         UsingSpan | .NET Framework 4.8 | .NET Framework 4.8 |         NA |        NA |        NA |

On the other hand, if you If order the frameworks so that net48 is first as <TargetFrameworks>net48;net6.0</TargetFrameworks> then (in the example above) the UsingSpan() method is excluded for both targets, resulting output such as this:

|            Method |                Job |            Runtime |     Mean |    Error |   StdDev |
|------------------ |------------------- |------------------- |---------:|---------:|---------:|
| UsingBitConverter |           .NET 6.0 |           .NET 6.0 | 343.1 us |  6.51 us | 11.57 us |
|      UsingMarshal |           .NET 6.0 |           .NET 6.0 | 539.5 us | 10.77 us | 22.94 us |
| UsingBitConverter | .NET Framework 4.8 | .NET Framework 4.8 | 331.2 us |  5.43 us |  5.08 us |
|      UsingMarshal | .NET Framework 4.8 | .NET Framework 4.8 | 588.9 us | 11.18 us | 10.98 us |    

I have to solve this issue by single-targeting the project and editing the project file to target the frameworks separately, and then run the benchmarks separately for each target.

Is there a way to make this work with a multi-targeted project?

---

For completeness, here's a full compilable test app which demonstrates the issue. I'm using Visual Studio 2022.

The project file:

<PropertyGroup>
  <OutputType>Exe</OutputType>
  <TargetFrameworks>net48;net6.0</TargetFrameworks>
  <ImplicitUsings>enable</ImplicitUsings>
  <LangVersion>latest</LangVersion>
  <Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
  <PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
</ItemGroup>

The "Program.cs" file:

using System.Runtime.InteropServices;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace Benchmark;

public static class Program
{
    public static void Main()
    {
        BenchmarkRunner.Run<UnderTest>();
    }
}

[SimpleJob(RuntimeMoniker.Net48)]
[SimpleJob(RuntimeMoniker.Net60)]
public class UnderTest
{
    [Benchmark]
    public void UsingBitConverter()
    {
        using var stream = new MemoryStream();
        writeUsingBitConverter(stream, _array);
    }

    static void writeUsingBitConverter(Stream output, double[] array)
    {
        foreach (var sample in array)
        {
            output.Write(BitConverter.GetBytes(sample), 0, sizeof(double));
        }
    }

    [Benchmark]
    public void UsingMarshal()
    {
        using var stream = new MemoryStream();
        writeUsingMarshal(stream, _array);
    }

    static void writeUsingMarshal(Stream output, double[] array)
    {
        const int SIZE_BYTES = sizeof(double);

        byte[] buffer = new byte[SIZE_BYTES];
        IntPtr ptr    = Marshal.AllocHGlobal(SIZE_BYTES);

        foreach (var sample in array)
        {
            Marshal.StructureToPtr(sample, ptr, true);
            Marshal.Copy(ptr, buffer, 0, SIZE_BYTES);
            output.Write(buffer, 0, SIZE_BYTES);
        }

        Marshal.FreeHGlobal(ptr);
    }

    #if NET6_0_OR_GREATER

    [Benchmark]
    public void UsingSpan()
    {
        using var stream = new MemoryStream();
        writeUsingSpan(stream, _array);
    }

    static void writeUsingSpan(Stream output, double[] array)
    {
        var span  = array.AsSpan();
        var bytes = MemoryMarshal.AsBytes(span);

        output.Write(bytes);
    }

    #endif // NET6_0_OR_GREATER

    readonly double[] _array = new double[10_000];
}

Answer 1

From memory, Benchmark.NET will run benchmarks for all frameworks with some internal wizardry. So instead of using the existing preprocessor symbols it's probably better to split your tests across two classes with different RuntimeMoniker attributes. For example:

[SimpleJob(RuntimeMoniker.Net48)]
public class UnderTestNet48
{
    // Benchmarks
}

[SimpleJob(RuntimeMoniker.Net60)]
public class UnderTestNet60
{
    // Benchmarks
}

And now you need to modify the code that runs the benchmarks since the they are split across classes, something like this will work:

public static void Main()
{
    var config = DefaultConfig.Instance.
        .WithOptions(ConfigOptions.JoinSummary)
        .WithOptions(ConfigOptions.DisableLogFile);

    BenchmarkRunner.Run(typeof(Program).Assembly, config);
}

---

[EDIT from OP (Matthew Watson)]

I managed to implement this thanks to this answer.

I managed to reduce the code duplication a bit by putting the common test methods into a protected base class, then providing two derived classes - one for the net48 benchmark and one for the net5.0 benchmark.

This is the code I ended up with:

using System.Runtime.InteropServices;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace Benchmark;

public static class Program
{
    public static void Main()
    {
        BenchmarkRunner.Run(
            typeof(Program).Assembly, 
            DefaultConfig.Instance
               .WithOptions(ConfigOptions.JoinSummary)
               .WithOptions(ConfigOptions.DisableLogFile));
    }
}

public abstract class UnderTestBase
{
    protected static Stream CreateStream()
    {
        return new MemoryStream(); // Or Stream.Null
    }

    protected void WriteUsingBitConverter(Stream output, double[] array)
    {
        foreach (var sample in array)
        {
            output.Write(BitConverter.GetBytes(sample), 0, sizeof(double));
        }
    }

    protected void WriteUsingMarshal(Stream output, double[] array)
    {
        const int SIZE_BYTES = sizeof(double);

        byte[] buffer = new byte[SIZE_BYTES];
        IntPtr ptr    = Marshal.AllocHGlobal(SIZE_BYTES);

        foreach (var sample in array)
        {
            Marshal.StructureToPtr(sample, ptr, true);
            Marshal.Copy(ptr, buffer, 0, SIZE_BYTES);
            output.Write(buffer, 0, SIZE_BYTES);
        }

        Marshal.FreeHGlobal(ptr);
    }

    #if NET6_0_OR_GREATER
    
    protected void WriteUsingSpan(Stream output, double[] array)
    {
        var span  = array.AsSpan();
        var bytes = MemoryMarshal.AsBytes(span);

        output.Write(bytes);
    }

    #endif // NET6_0_OR_GREATER

    protected readonly double[] Array = new double[100_000];
}

[SimpleJob(RuntimeMoniker.Net48)]
public class UnderTestNet48: UnderTestBase
{
    [Benchmark]
    public void UsingBitConverter()
    {
        using var stream = CreateStream();
        WriteUsingBitConverter(stream, Array);
    }

    [Benchmark]
    public void UsingMarshal()
    {
        using var stream = CreateStream();
        WriteUsingMarshal(stream, Array);
    }
}

[SimpleJob(RuntimeMoniker.Net60)]
public class UnderTestNet60: UnderTestBase
{
    [Benchmark]
    public void UsingBitConverter()
    {
        using var stream = CreateStream();
        WriteUsingBitConverter(stream, Array);
    }

    [Benchmark]
    public void UsingMarshal()
    {
        using var stream = CreateStream();
        WriteUsingMarshal(stream, Array);
    }

    #if NET6_0_OR_GREATER

    [Benchmark]
    public void UsingSpan()
    {
        using var stream = CreateStream();
        WriteUsingSpan(stream, Array);
    }

    #endif // NET6_0_OR_GREATER
}

Which results in this output:

|           Type |            Method |                Job |            Runtime |       Mean |     Error |    StdDev |
|--------------- |------------------ |------------------- |------------------- |-----------:|----------:|----------:|
| UnderTestNet60 | UsingBitConverter |           .NET 6.0 |           .NET 6.0 | 4,110.8 us |  81.53 us | 151.13 us |
| UnderTestNet60 |      UsingMarshal |           .NET 6.0 |           .NET 6.0 | 5,774.0 us | 114.78 us | 194.90 us |
| UnderTestNet60 |         UsingSpan |           .NET 6.0 |           .NET 6.0 |   521.6 us |   5.13 us |   4.80 us |
| UnderTestNet48 | UsingBitConverter | .NET Framework 4.8 | .NET Framework 4.8 | 2,987.2 us |  35.60 us |  29.73 us |
| UnderTestNet48 |      UsingMarshal | .NET Framework 4.8 | .NET Framework 4.8 | 5,616.9 us |  57.85 us |  48.30 us |

(As an aside, one interesting result is that the UsingBitConverter() method actually seems to run faster with net48 compared to net6.0 - although this is dwarfed by the massive improvement in speed offered by Span<T>.)

[/EDIT from OP (Matthew Watson)]

Answer 2

It has been discussed in https://github.com/dotnet/BenchmarkDotNet/issues/1226#issuecomment-532144829:

When running the host process targetting XYZ framework, BDN is using the reflection to obtain the list of available methods (benchmarks). If you are using #if defines, then the list of benchmarks is going to be different per host process target framework.

Performance repo docs describe how to compare multiple runtimes performance here: https://github.com/dotnet/performance/blob/master/docs/benchmarkdotnet.md#multiple-runtimes

The host process needs to be the lowest common API denominator of the runtimes you want to compare!

You can read more about benchmarking multiple tfms in https://benchmarkdotnet.org/articles/configs/toolchains.html#multiple-frameworks-support