15

I am reading "C# via CLR" and on page 380, there's a note saying the following:

Note The Enum class defines a HasFlag method defined as follows

public Boolean HasFlag(Enum flag);

Using this method, you could rewrite the call to Console.WriteLine like this:

Console.WriteLine("Is {0} hidden? {1}", file, attributes.HasFlag(FileAttributes.Hidden));

However, I recommend that you avoid the HasFlag method for this reason:

Since it takes a parameter of type Enum, any value you pass to it must be boxed, requiring a memory allocation ."

I can not understand this bolded statement -- why "

any value you pass to it must be boxed

The flag parameter type is Enum, which is a value type, why would there be boxing? The "any value you pass to it must be boxed" should mean boxing happens when you pass value type to parameter Enum flag, right?

Marcel Gosselin
  • 4,610
  • 2
  • 31
  • 54
user1553932
  • 153
  • 1
  • 6
  • It all comes down to a single, but confusing, statement: `Enum` is not an *enum*... – Marc Gravell Jul 26 '12 at 09:00
  • @MarcGravell Indeed, I've spent a long chain of comments trying to defend my answer from the fact that people refuse to believe that statement. Confused by: `ValueType` is not a value type lol... – Adam Houldsworth Jul 26 '12 at 09:03
  • Note that as of .NET Core 2.1, `Enum.HasFlag` doesn't box I believe: https://blogs.msdn.microsoft.com/dotnet/2018/04/18/performance-improvements-in-net-core-2-1. While I could see `box` instruction in IL still in 2.1 app, it doesn't allocate, hence I dont see the perf penalty. – nawfal Oct 24 '18 at 06:44

8 Answers8

9

In this instance, two boxing calls are required before you even get into the HasFlags method. One is for resolving the method call on the value type to the base type method, the other is passing the value type as a reference type parameter. You can see the same in IL if you do var type = 1.GetType();, the literal int 1 is boxed before the GetType() call. The boxing on method call seems to be only when methods are not overridden in the value type definition itself, more can be read here: Does calling a method on a value type result in boxing in .NET?

The HasFlags takes an Enum class argument, so the boxing will occur here. You are trying to pass what is a value type into something expecting a reference type. To represent values as references, boxing occurs.

There is lots of compiler support for value types and their inheritance (with Enum / ValueType) that confuses the situation when trying to explain it. People seem to think that because Enum and ValueType is in the inheritance chain of value types boxing suddenly doesn't apply. If this were true, the same could be said of object as everything inherits that - but as we know this is false.

This doesn't stop the fact that representing a value type as a reference type will incur boxing.

And we can prove this in IL (look for the box codes):

class Program
{
    static void Main(string[] args)
    {
        var f = Fruit.Apple;
        var result = f.HasFlag(Fruit.Apple);

        Console.ReadLine();
    }
}

[Flags]
enum Fruit
{
    Apple
}



.method private hidebysig static 
    void Main (
        string[] args
    ) cil managed 
{
    // Method begins at RVA 0x2050
    // Code size 28 (0x1c)
    .maxstack 2
    .entrypoint
    .locals init (
        [0] valuetype ConsoleApplication1.Fruit f,
        [1] bool result
    )

    IL_0000: nop
    IL_0001: ldc.i4.0
    IL_0002: stloc.0
    IL_0003: ldloc.0
    IL_0004: box ConsoleApplication1.Fruit
    IL_0009: ldc.i4.0
    IL_000a: box ConsoleApplication1.Fruit
    IL_000f: call instance bool [mscorlib]System.Enum::HasFlag(class [mscorlib]System.Enum)
    IL_0014: stloc.1
    IL_0015: call string [mscorlib]System.Console::ReadLine()
    IL_001a: pop
    IL_001b: ret
} // end of method Program::Main

The same can be seen when representing a value type as ValueType, it also results in boxing:

class Program
{
    static void Main(string[] args)
    {
        int i = 1;
        ValueType v = i;

        Console.ReadLine();
    }
}


.method private hidebysig static 
    void Main (
        string[] args
    ) cil managed 
{
    // Method begins at RVA 0x2050
    // Code size 17 (0x11)
    .maxstack 1
    .entrypoint
    .locals init (
        [0] int32 i,
        [1] class [mscorlib]System.ValueType v
    )

    IL_0000: nop
    IL_0001: ldc.i4.1
    IL_0002: stloc.0
    IL_0003: ldloc.0
    IL_0004: box [mscorlib]System.Int32
    IL_0009: stloc.1
    IL_000a: call string [mscorlib]System.Console::ReadLine()
    IL_000f: pop
    IL_0010: ret
} // end of method Program::Main
Community
  • 1
  • 1
Adam Houldsworth
  • 63,413
  • 11
  • 150
  • 187
  • Yes, you are right, but in this case the boxing in occurring due to the call the Console.WriteLine. Jeff Richter has a large section in the book about avoiding boxing, and I believe this is where it comes from. – stevethethread Jul 26 '12 at 08:41
  • @stevethethread So why are there boxing IL commands in my sample? – Adam Houldsworth Jul 26 '12 at 08:41
  • 1
    This isn't answer, this just repeats the observation leading to this question. –  Jul 26 '12 at 08:42
  • 1
    @hvd It explains why there is boxing, [`Enum` is a class...](http://msdn.microsoft.com/en-us/library/system.enum(v=vs.100).aspx) – Adam Houldsworth Jul 26 '12 at 08:42
  • @AdamHouldsworth ...which derives from `ValueType`. –  Jul 26 '12 at 08:43
  • @hvd The question doesn't care about that. Though the mechanics of the true answer might. The `ValueType` base class is handled specially by the compiler, I've no idea what handling `enum` the keyword gets. – Adam Houldsworth Jul 26 '12 at 08:43
  • @AdamHouldsworth From the question: "flag parament type is Enum, which is value type". You're basically saying `Enum` derives from `ValueType`, but isn't a value type. You may be right, but that could use some more highlighting. –  Jul 26 '12 at 08:45
  • Actually just read the section in the book, and he is indeed referring to the call to the Enum.HasFlag method. My mistake. – stevethethread Jul 26 '12 at 08:45
  • 1
    @hvd Yes the documentation for Enum, enum, and ValueType state the inheritance chains but offer nothing as to how this is realised by the compiler. – Adam Houldsworth Jul 26 '12 at 08:46
  • I think adam should be right. I confuse Enum type and enum type – user1553932 Jul 26 '12 at 08:47
  • Lowercase enum isn't a type--it's a keyword. (Try `typeof(enum)` vs `typeof(Enum)` vs `typeof(int)` if you need to convince yourself.) – sblom Jul 26 '12 at 08:50
  • This answer seriously isn't the answer. It's more information on the path to the answer, but we're not there yet. -1 (for now?) – sblom Jul 26 '12 at 08:51
  • 1
    @sblom -1 for correct information? My final statement is my answer. The remaining question is why? Which isn't being asked. `int` inherits `object`, so would you expect boxing to occur there? – Adam Houldsworth Jul 26 '12 at 08:51
  • @AdamHouldsworth, There is _tons_ of misinformation here. `int` and `Enum` have the same type hierarchy. Try `Console.WriteLine(typeof(int).BaseType == typeof(Enum).BaseType && typeof(int).BaseType == typeof(ValueType));` if you need proof. – sblom Jul 26 '12 at 08:56
  • 2
    @sblom The misinformation is **only** in people implying that inheriting from `Enum` or `ValueType` means that representing these values types in either `Enum` or `ValueType` shouldn't incur boxing... simply, it does. The same argument could be made for `object`. The answer is truly as simple as it can get: it boxes because you are representing a value type as a reference type. – Adam Houldsworth Jul 26 '12 at 08:57
  • @AdamHouldsworth: It's too bad that `Enum` wasn't implemented as a `struct` containing an `Int64` and a `Type`. Such a design might not have nicely supported `Enum` types backed by a `Uint64` [how often are those used?] but could otherwise have allowed easy non-boxing widening conversions to `Enum` from any enumerated type, and narrowing conversions from `Enum` to any enumerated type (the enumerated types themselves would only need to be 1, 2, 4, or 8-byte type, since each `Enum` type would know what its own type was). – supercat Oct 07 '14 at 20:21
9

It's worth noting that a generic HasFlag<T>(T thing, T flags) which is about 30 times faster than the Enum.HasFlag extension method can be written in about 30 lines of code. It can even be made into an extension method. Unfortunately, it's not possible in C# to restrict such a method to only take things of enumerated types; consequently, Intellisense will pop up the method even for types for which it is not applicable. I think if one used some language other than C# or vb.net to write the extension method it might be possible to make it pop up only when it should, but I'm not familiar enough with other languages to try such a thing.

internal static class EnumHelper<T1>
{
    public static Func<T1, T1, bool> TestOverlapProc = initProc;
    public static bool Overlaps(SByte p1, SByte p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(Byte p1, Byte p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(Int16 p1, Int16 p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(UInt16 p1, UInt16 p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(Int32 p1, Int32 p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(UInt32 p1, UInt32 p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(Int64 p1, Int64 p2) { return (p1 & p2) != 0; }
    public static bool Overlaps(UInt64 p1, UInt64 p2) { return (p1 & p2) != 0; }
    public static bool initProc(T1 p1, T1 p2)
    {
        Type typ1 = typeof(T1);
        if (typ1.IsEnum) typ1 = Enum.GetUnderlyingType(typ1);
        Type[] types = { typ1, typ1 };
        var method = typeof(EnumHelper<T1>).GetMethod("Overlaps", types);
        if (method == null) method = typeof(T1).GetMethod("Overlaps", types);
        if (method == null) throw new MissingMethodException("Unknown type of enum");
        TestOverlapProc = (Func<T1, T1, bool>)Delegate.CreateDelegate(typeof(Func<T1, T1, bool>), method);
        return TestOverlapProc(p1, p2);
    }
}
static class EnumHelper
{
    public static bool Overlaps<T>(this T p1, T p2) where T : struct
    {
        return EnumHelper<T>.TestOverlapProc(p1, p2);
    }
}

EDIT: A previous version was broken, because it used (or at least tried to use) EnumHelper<T1, T1>.

mike
  • 1,627
  • 1
  • 14
  • 37
supercat
  • 77,689
  • 9
  • 166
  • 211
  • That's some magic! going to steal it :) I think you should definitely make it an answer here: http://stackoverflow.com/questions/9519596/hasflag-with-a-generic-enum – nawfal Jun 21 '13 at 07:57
  • 2
    Caching `method` will make it even faster. – IS4 Jun 03 '15 at 07:42
  • 2
    You can apply the System.Enum restrain in [C# 7.3](https://stackoverflow.com/a/8086788/9453080)! – Ivan García Topete Sep 03 '19 at 17:16
  • @IvanGarcíaTopete If you're stuck with an older version as I am, you can use `struct` as a constraint and with that at least prevent Intellisense from offering the extension method for basically _anything_. – mike Nov 28 '20 at 13:48
  • 2
    @supercat: Despite being allocation free, your solution is twice as slow as the native `HasFlag` in my tests (calling each 10,000,000 times in a loop in a unit test). Could it be that calling the delegate outweighs the costs of boxing? Or is my unit test misleading? – mike Nov 28 '20 at 14:30
  • @supercat Why the second check, `if (method == null) method = typeof(T1).GetMethod("Overlaps", types);`? – mike Nov 28 '20 at 14:35
  • @mike: In 2012, the HasFlag method would generally box not only the object upon which it was invoked, but also the the object it was being tested against. Additionally, each invocation would need to use Reflection to determine how the values were represented. My method is much slower than the ideal machine code to test flags, but was faster than the way `HasFlag` actually worked. It may well be that in newer versions of C# and .NET the performance of `HasFlag` has been improved to be better than my version, but I've not used .NET lately except for maintaining existing projects. – supercat Nov 28 '20 at 17:23
  • @mike: When the code was written, `System.Enum` was not available as a generic constraint, and the extension method could be used on a reference of type `System.Enum` or `System.Object`. If invocation with a generic type parameter of `System.Object` were to and cached a lookup method for `System.Object`, then an attempt to later use the method on a different kind of object would use an inappropriate cached method. – supercat Nov 28 '20 at 17:31
  • 1
    @supercat I think are mistakenly _not_ caching the delegate. That explains why @mike has reported your solution as slower than the native `GetFlag`. At the time I'm writing this: `TestOverlapProc = initProc` makes the former simply a reference to the latter. The latter is a method that, _whenever invoked_, builds a delegate (a costly operation that should not be repeated), runs it, and returns the result. Instead, you could build the delegate once and store it. I'm guessing this may have been your intent. – Timo Aug 02 '21 at 14:05
  • @Timo The first call to `TestOverlapProc` will call `initProc`, which creates the correct `Overlaps` delegate to use, and then updates `TestOverlapProc` to point to `Overlaps`, and finally calls it. So the 2nd time you call `TestOverlapProc` it will directly call the right `Overlaps` method. – user276648 Dec 01 '21 at 00:52
  • 1
    O _wow_, that is obscure! `TestOverlapProc` gets initialized to point to `initProc`, which, on first invocation, creates the delegate, _overwrites_ `TestOverlapProc`, _and_ invokes it to get the result (performing multiple responsibilities and belying its name). Subsequent invocations then lead to the new value, i.e. the compiled delegate. That is horribly cryptic. A static initializer or simple `??=` would have avoided the confusion. – Timo Dec 02 '21 at 20:42
  • @Timo: The code is intended to minimize runtime overhead for all but the first invocation of `Overlaps` for any particular type. The pattern used here isn't universally known, but it's widely used in many kinds of lazy code creation or loading such as e.g. 8087 floating-point emulation. Initially, a compiler will place an INT instruction where a floating-point instruction would go. The interrupt handler will then, if an FPU is present, replace the instruction with the appropriate FPU instruction, subtract 2 from the return address, and resume program execution. – supercat Dec 02 '21 at 21:29
4

Enum inherits from ValueType which is... a class! Hence the boxing.

Note that the Enum class can represents any enumeration, whatever its underlying type is, as a boxed value. Whereas a value such as FileAttributes.Hidden will be represented as real value type, int.

Edit: let's differentiate the type and the representation here. An int is represented in memory as 32 bits. Its type derives from ValueType. As soon as you assign an int to an object or derived class (ValueType class, Enum class), you're boxing it, effectively changing its representation to a class now containing that 32 bits, plus additional class information.

Julien Lebosquain
  • 40,639
  • 8
  • 105
  • 117
  • I don't understand your first sentence. `void f(int i) { } void g() { f(3); }` -- `int` also inherits from `ValueType`, but there is no boxing there. Same if `int` is changed to a concrete enum type. –  Jul 26 '12 at 08:33
  • 1
    That can't be the whole story. `System.Int32` inherits from `ValueType`, too. – sblom Jul 26 '12 at 08:34
  • Yes, the method here takes an int, not object. – stevethethread Jul 26 '12 at 08:34
  • 3
    @JulienLebosquain Yes, it does. All structs inherit from `ValueType`. –  Jul 26 '12 at 08:35
  • 1
    @JulienLebosquain See the struct page on msdn http://msdn.microsoft.com/en-us/library/saxz13w4.aspx :: A struct cannot inherit from another struct or class, and it cannot be the base of a class. All structs inherit directly from System.ValueType, which inherits from System.Object. – SynerCoder Jul 26 '12 at 08:42
  • 1
    Edited my post, trying to explain the difference between a Type and its value representation. – Julien Lebosquain Jul 26 '12 at 08:42
  • @hvd: Storage locations of types which inherit from `System.ValueType` or `System.Enum`, *but are not of those two exact types*`, are value types. Storage locations of all other types that derive from `Object` are class types; that includes `System.ValueType` and `System.Enum` themselves. – supercat Dec 18 '12 at 17:39
4

Since C# 7.3, where generic Enum constraint was introduced, you can write a fast, non allocating version that doesn't rely on reflection. It requires the compiler flag /unsafe but since Enum backing types can only be a fixed amount of sizes, it should be perfectly safe to do:

using System;
using System.Runtime.CompilerServices;
public static class EnumFlagExtensions
{
    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public static bool HasFlagUnsafe<TEnum>(TEnum lhs, TEnum rhs) where TEnum : unmanaged, Enum
    {
        unsafe
        {
            switch (sizeof(TEnum))
            {
                case 1:
                    return (*(byte*)(&lhs) & *(byte*)(&rhs)) > 0;
                case 2:
                    return (*(ushort*)(&lhs) & *(ushort*)(&rhs)) > 0;
                case 4:
                    return (*(uint*)(&lhs) & *(uint*)(&rhs)) > 0;
                case 8:
                    return (*(ulong*)(&lhs) & *(ulong*)(&rhs)) > 0;
                default:
                    throw new Exception("Size does not match a known Enum backing type.");
            }
        }
    }
}
Martin Tilo Schmitz
  • 176
  • 1
  • 5
  • `Unsafe.As()` should allow for an implementation that does not require the `/unsafe` compiler flag. E.g. `Unsafe.As(ref lhs)`. _(The same goes for `MemoryMarshal.Cast()`, although we would first have to turn the value into a span using `MemoryMarshal.CreateSpan()`.)_ – Timo Sep 16 '21 at 07:54
1

As suggested by Timo the solution of Martin Tilo Schmitz can be implemented without the need for the /unsafe switch:

public static bool HasAnyFlag<E>(this E lhs, E rhs) where E : unmanaged, Enum
{
    switch (Unsafe.SizeOf<E>())
    {
    case 1:
        return (Unsafe.As<E, byte>(ref lhs) & Unsafe.As<E, byte>(ref rhs)) != 0;
    case 2:
        return (Unsafe.As<E, ushort>(ref lhs) & Unsafe.As<E, ushort>(ref rhs)) != 0;
    case 4:
        return (Unsafe.As<E, uint>(ref lhs) & Unsafe.As<E, uint>(ref rhs)) != 0;
    case 8:
        return (Unsafe.As<E, ulong>(ref lhs) & Unsafe.As<E, ulong>(ref rhs)) != 0;
    default:
        throw new Exception("Size does not match a known Enum backing type.");
    }
}

The NuGet System.Runtime.CompilerServices.Unsafe is required to compile this with .NET Framework.

Performance

  • I have to mention that any generic implementation is still almost an order of magnitude slower than a native check, i.e. ((int)lhs & (int)rhs) != 0. I guess taking the reference of lhs, rhs prevents optimization of the storage of the function variables. The runtime dispatch of the enum size adds another overhead.
  • But it is still one order of magnitude faster than HasFlag.
  • And well, the performance benefit is almost zero comapred to HasFlag if optimizations are turned off in a debug build.
  • There is no significant difference between using unsafe { } and using class Unsafe in optimized (release) builds. Only without optimizations class Unsafe is almost as slow as HasFlag.
  • Using a delegate as supercat recommended is no reasonable option since the resulting function pointer call is slow on most CPU architectures and even more it completely prevents inlining.
  • MethodImplOptions.AggressiveInlining adds no value.

Conclusion

There is still no really fast and readable implementation for testing flags in enums.

Marcel
  • 1,688
  • 1
  • 14
  • 25
  • i would like to add: using tests to evaluate the real-world performance impact is often hard (if not impossible) when it comes to heap allocation and garbage collection. the safest way (albeit not prettiest) is sadly still `a & b != 0`... – mike Jul 31 '22 at 23:34
0

When ever you pass a value type of a method that takes object as a parameter, as in the case of console.writeline, there will be an inherent boxing operation. Jeffery Richter discusses this in detail in the same book you mention.

In this case you are using the string.format method of console.writeline, and that takes a params array of object[]. So your bool, will be cast to object, so hence you get a boxing operation. You can avoid this by calling .ToString() on the bool.

stevethethread
  • 2,524
  • 2
  • 30
  • 29
  • `Enum.HasFlag`'s parameter type isn't `object`. –  Jul 26 '12 at 08:34
  • That's boxing the `bool` result of `Enum.HasFlag()`, not the argument to `HasFlag()`... – sblom Jul 26 '12 at 08:35
  • Enum.HasFlag() returns bool, a value type, and so, boxing. – stevethethread Jul 26 '12 at 08:37
  • @stevethethread That isn't what this question is about, and besides, returning a value type doesn't require boxing unless the function is declared as returning a reference type (`object`). –  Jul 26 '12 at 08:38
  • but here "any value you pass to it must be boxed" should mean boxing happens when you pass value type to parameter "Enum flag", right? – user1553932 Jul 26 '12 at 08:40
0

Moreover, there's more than single boxing in Enum.HasFlag:

public bool HasFlag(Enum flag)
{
    if (!base.GetType().IsEquivalentTo(flag.GetType()))
    {
        throw new ArgumentException(Environment.GetResourceString("Argument_EnumTypeDoesNotMatch", new object[]
        {
            flag.GetType(),
            base.GetType()
        }));
    }
    ulong num = Enum.ToUInt64(flag.GetValue());
    ulong num2 = Enum.ToUInt64(this.GetValue());
    return (num2 & num) == num;
}

Look at GetValue method calls.

Update. Looks like MS had optimized this method in .NET 4.5 (the source code has been downloaded from referencesource):

    [System.Security.SecuritySafeCritical]
    public Boolean HasFlag(Enum flag) { 
        if (flag == null)
            throw new ArgumentNullException("flag"); 
        Contract.EndContractBlock(); 

        if (!this.GetType().IsEquivalentTo(flag.GetType())) { 
            throw new ArgumentException(Environment.GetResourceString("Argument_EnumTypeDoesNotMatch", flag.GetType(), this.GetType()));
        }

        return InternalHasFlag(flag); 
    }

    [System.Security.SecurityCritical]  // auto-generated 
    [ResourceExposure(ResourceScope.None)]
    [MethodImplAttribute(MethodImplOptions.InternalCall)] 
    private extern bool InternalHasFlag(Enum flags);
Dennis
  • 37,026
  • 10
  • 82
  • 150
  • Is that the actual implementation? Seems needlessly slow. It's possible, and not overly hard, to write a static method `bool HasFlag(T p1, T p2)` which will run about more than 10x as fast as `enum.HasFlag`. – supercat Dec 18 '12 at 17:37
  • @supercat: excellent question, indeed. It is actual for .NET 4.0, for .NET 4.5 it is different. See updated answer. – Dennis Dec 19 '12 at 05:33
  • I wonder how that affects performance? Using .net 4.0, my generic method seems about 6x slower than using `&` but 30x faster than `Enum.HasFlag`. Actually, I would think that testing an enumeration for flag values would be a sufficiently frequent operation that would be worthy of language support, especially since a language could separate out `HasAny`, `HasAll`, and `Has` cases, restricting the latter to operands that were constant powers of two [since it's otherwise ambiguous whether `SomeEnum.Has(3)` should mean `(SomeEnum & 3) != 0` or `(SomeEnum & 3)==3`.] – supercat Dec 19 '12 at 15:46
  • See my answer for the my code, if you'd like to benchmark it against the .net 4.5 version of HasFlag. (Incidentally, as shown, my version will work with any struct type `T` that defines an `Overlaps(T,T)` overload; not sure that's useful, but it shouldn't affect the benchmarks since each type is only evaluated once). – supercat Dec 19 '12 at 15:51
0

There are two boxing operations involved in this call, not just one. And both are required for one simple reason: Enum.HasFlag() needs type information, not just values, for both this and flag.

Most of the time, an enum value truly is just a set of bits and the compiler has all the type information it needs from the enum types represented in the method signature.

However, in the case of Enum.HasFlags() the very first thing it does is call this.GetType() and flag.GetType() and make sure they're identical. If you wanted the typeless version, you'd be asking if ((attribute & flag) != 0), instead of calling Enum.HasFlags().

sblom
  • 26,911
  • 4
  • 71
  • 95
  • Some time passed but anyway: this is not true. If you ask for `GetType()` inside some method then boxing will occur just inside this method. You could easily test it with simple value type with some method which call `GetType()` for example. You'll see that boxing will occur inside your method, not when you calling it from some outer code. – iw.kuchin Oct 09 '12 at 05:13
  • @iw.kuchin: The type `System.Enum` is a class type, as is the ironically-named `System.ValueType`. Calling `Enum.HasFlag(Enum)` requires casting its argument to `System.Enum`, which means it will be boxed before the `HasFlag` method gets a chance to execute. – supercat Dec 18 '12 at 17:30
  • @supercat Yes, and this is the real reason because boxing occurs here. Not because type information is needed somewhere inside Enum.HasFlag(). – iw.kuchin Dec 20 '12 at 09:36