Why does LayoutKind.Sequential work differently if a struct contains a DateTime field?

Question

Consider the following code (a console app which must be compiled with "unsafe" enabled):

using System;
using System.Runtime.InteropServices;

namespace ConsoleApplication3
{
    static class Program
    {
        static void Main()
        {
            Inner test = new Inner();

            unsafe
            {
                Console.WriteLine("Address of struct   = " + ((int)&test).ToString("X"));
                Console.WriteLine("Address of First    = " + ((int)&test.First).ToString("X"));
                Console.WriteLine("Address of NotFirst = " + ((int)&test.NotFirst).ToString("X"));
            }
        }
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct Inner
    {
        public byte First;
        public double NotFirst;
        public DateTime WTF;
    }
}

Now if I run the code above, I get output similar to the following:

Address of struct = 40F2CC
Address of First = 40F2D4
Address of NotFirst = 40F2CC

Note that the address of First is NOT the same as the address of the struct; however, the address of NotFirst is the same as the address of the struct.

Now comment out the "DateTime WTF" field in the struct, and run it again. This time, I get output similar to this:

Address of struct = 15F2E0
Address of First = 15F2E0
Address of NotFirst = 15F2E8

Now "First" does have the same address as the struct.

I find this behaviour surprising given the use of LayoutKind.Sequential. Can anyone provide an explanation? Does this behaviour have any ramifications when doing interop with C/C++ structs that use the Com DATETIME type?

[EDIT] NOTE: I have verified that when you use Marshal.StructureToPtr() to marshal the struct, the data is marshalled in the correct order, with the "First" field being first. This seems to suggest that it will work fine with interop. The mystery is why the internal layout changes - but of course, the internal layout is never specified, so the compiler can do what it likes.

[EDIT2] Removed "unsafe" from struct declaration (it was leftover from some testing I was doing).

[EDIT3] The original source for this question was from the MSDN C# forums:

http://social.msdn.microsoft.com/Forums/en-US/csharplanguage/thread/fb84bf1d-d9b3-4e91-823e-988257504b30

Well thank goodness one never has to use DateTime when going unsafe. :) — leppie, Nov 09 '10 at 10:32
+1 for answering your question. You should create an answer with your own answer and accept it when you can. — jgauffin, Nov 09 '10 at 10:46
I don't think it is valid to try to include a datetime because it contains string data internally. see http://social.msdn.microsoft.com/Forums/en/clr/thread/2d40ad89-3510-4cfb-9695-435e38174421 for more — Kell, Nov 09 '10 at 10:51
@Kell: Static members do not affect the layout, and that is the only place with string is used. — leppie, Nov 09 '10 at 10:59
@Kell: Also, the Marshalling stuff seems to handle DateTime specially, which might be related to this "interesting" behaviour. :) See http://msdn.microsoft.com/en-us/library/0t2cwe11%28VS.71%29.aspx — Matthew Watson, Nov 09 '10 at 14:02
Ah, the MS way of doing things: we are special and special rules apply to our classes :) — Kell, Nov 09 '10 at 14:27
Two remarks: (1) It is actually redundant (but maybe informative?) to specify `LayoutKind.Sequential` since this is the default in C# for a `struct` type. (2) This "problem" may be related to the [fact that `DateTime` itself has layout "Auto"](http://stackoverflow.com/questions/21881554/), since if you use `TimeSpan` (another struct of the same size) instead of `DateTime` as the type of your field `WTF`, the "problem" goes away. — Jeppe Stig Nielsen, Feb 19 '14 at 14:11

score 20 · Accepted Answer · edited May 23 '17 at 11:47

Why does LayoutKind.Sequential work differently if a struct contains a DateTime field?

It is related to the (surprising) fact that DateTime itself has layout "Auto" (link to SO question by myself). This code reproduces the behavior you saw:

static class Program
{
    static unsafe void Main()
    {
        Console.WriteLine("64-bit: {0}", Environment.Is64BitProcess);
        Console.WriteLine("Layout of OneField: {0}", typeof(OneField).StructLayoutAttribute.Value);
        Console.WriteLine("Layout of Composite: {0}", typeof(Composite).StructLayoutAttribute.Value);
        Console.WriteLine("Size of Composite: {0}", sizeof(Composite));
        var local = default(Composite);
        Console.WriteLine("L: {0:X}", (long)(&(local.L)));
        Console.WriteLine("M: {0:X}", (long)(&(local.M)));
        Console.WriteLine("N: {0:X}", (long)(&(local.N)));
    }
}

[StructLayout(LayoutKind.Auto)]  // also try removing this attribute
struct OneField
{
    public long X;
}

struct Composite   // has layout Sequential
{
    public byte L;
    public double M;
    public OneField N;
}

Sample output:

64-bit: True
Layout of OneField: Auto
Layout of Composite: Sequential
Size of Composite: 24
L: 48F050
M: 48F048
N: 48F058

If we remove the attribute from OneField, things behave as expected. Example:

64-bit: True
Layout of OneField: Sequential
Layout of Composite: Sequential
Size of Composite: 24
L: 48F048
M: 48F050
N: 48F058

These example are with x64 platform compilation (so the size 24, three times eight, is unsurprising), but also with x86 we see the same "disordered" pointer addresses.

So I guess I can conclude that the layout of OneField (resp. DateTime in your example) has influence on the layout of the struct containing a OneField member even if that composite struct itself has layout Sequential. I am not sure if this is problematic (or even required).

According to comment by Hans Passant in the other thread, it no longer makes an attempt to keep it sequential when one of the members is an Auto layout struct.

Finally an answer to this question that actually makes sense. — CodesInChaos, Mar 03 '14 at 19:32

Jonathan S. Shapiro · Answer 2 · 2014-05-13T13:14:03.840

Go read the specification for layout rules more carefully. Layout rules only govern the layout when the object is exposed in unmanaged memory. This means that the compiler is free to place the fields however it wants until the object is actually exported. Somewhat to my surprise, this is even true for FixedLayout!

Ian Ringrose is right about compiler efficiency issues, and that does account for the final layout that is being selected here, but it has nothing to do with why the compiler is ignoring your layout specification.

A couple of people have pointed out that DateTime has Auto layout. That is the ultimate source of your surprise, but the reason is a bit obscure. The documentation for Auto layout says that "objects defined with [Auto] layout cannot be exposed outside of managed code. Attempting to do so generates an exception." Also note that DateTime is a value type. By incorporating a value type having Auto layout into your structure, you inadvertently promised that you would never expose the containing structure to unmanaged code (because doing so would expose the DateTime, and that would generate an exception). Since the layout rules only govern objects in unmanaged memory, and your object can never be exposed to unmanaged memory, the compiler is not constrained in its choice of layout and is free to do whatever it wants. In this case it is reverting to the Auto layout policy in order to achieve better structure packing and alignment.

There! Wasn't that obvious!

All of this, by the way, is recognizable at static compile time. In fact, the compiler is recognizing it in order to decide that it can ignore your layout directive. Having recognized it, a warning here from the compiler would seem to be in order. You haven't actually done anything wrong, but it's helpful to be told when you've written something that has no effect.

The various comments here recommending Fixed layout are generally good advice, but in this case that wouldn't necessarily have any effect, because including the DateTime field exempted the compiler from honoring layout at all. Worse: the compiler isn't required to honor layout, but it is free to honor layout. Which means that successive versions of CLR are free to behave differently on this.

The treatment of layout, in my view, is a design flaw in CLI. When the user specifies a layout, the compiler shouldn't go lawyering around them. Better to keep things simple and have the compiler do what it is told. Especially so where layout is concerned. "Clever", as we all know, is a four letter word.

Not a design flaw in CLI IMO. It's not supposed to change the layout of the managed structure at all - that's just a performance optimisation that allows the marshaller to avoid copying the structure in some cases. You're not setting the layout of the structure, you're instructing *the marshaller* to marshal the structure a certain way - and it *is* marshalled that way (in this case causing an exception). This is explicitly documented in https://msdn.microsoft.com/en-us/library/system.runtime.interopservices.structlayoutattribute(v=vs.110).aspx#Anchor_7 — Luaan, Feb 22 '17 at 13:06
The docs state about `ExplicitLayout`: _Use the attribute with LayoutKind.Explicit to control the precise position of each data member. This affects both managed and unmanaged layout, for both blittable and non-blittable types._ To me this means you do have control (as long as you don't use or include `Auto`). — Abel, Jun 02 '20 at 14:38
Because of the discussion and confusion here, I reported the differences in coverage in the docs and specs here: https://github.com/dotnet/dotnet-api-docs/issues/4325 — Abel, Jun 03 '20 at 15:41

Ian Ringrose · Answer 3 · 2010-11-09T11:21:04.373

3

A few factors

doubles are a lot faster if they are aligned
CPU caches may work better if there are no “holes” in the struck

So the C# compiler has a few undocumented rules it uses to try to get the “best” layout of structs, these rules may take into account the total size of a struct, and/or if it contains another struct etc. If you need to know the layout of a struct then you should specify it yourself rather than letting the compiler decide.

However the LayoutKind.Sequential does stop the compiler changing the order of the fields.

edited Nov 09 '10 at 11:21

answered Nov 09 '10 at 10:53

Ian Ringrose

51,220
55
213
317

1

So you just contradicted yourself? – leppie Nov 09 '10 at 11:27
1

@Leppie, no the docs for LayoutKind.Sequential say "... and can be noncontiguous" – Ian Ringrose Nov 09 '10 at 12:30
1

What about the fields changing order? Check the addresses in play here, not just their spacing, but their values/order as well. – Lasse V. Karlsen Nov 09 '10 at 13:32
It doesn't seem to be anything to do with the packing, since specifying Pack=1 doesn't change the ordering. – Matthew Watson Nov 09 '10 at 14:04
That is to do with packing, not ordering. – leppie Nov 09 '10 at 15:08
1

Another possible factor: `typeof(DateTime).IsAutoLayout` holds, and that is unusual. If you change to `TimeSpan`, say, since `typeof(TimeSpan).IsLayoutSequential`, you won't have that "anomaly". – Jeppe Stig Nielsen Feb 19 '14 at 14:19
2

There is an example in the BCL, namely `TimeZoneInfo.TransitionTime` (a.k.a. `System.TimeZoneInfo+TransitionTime`). Its fields, in order of declaration, and the other they are found with reflection, are: `DateTime m_timeOfDay; byte m_month; byte m_week; byte m_day; DayOfWeek m_dayOfWeek; bool m_isFixedDateRule;`. But when seen with `unsafe` pointers, they come in the order `|dayOfWeek|month|week|day|isFixedDateRule|timeOfDay|`. Those are sizes `4+1+1+1+1+8`. This `struct` has layout _sequential_, but its fields are not in order. – Jeppe Stig Nielsen Feb 20 '14 at 23:38
@JeppeStigNielsen `StructLayout` specifies the way the structure is marshalled, not its managed layout - that's just an implementation detail (a performance optimisation that allows the marshaller to avoid copying the structure in some interop scenarios). Taking pointers to fields in a struct is dealing with the managed layout, which isn't constrained by contract at all. Use `Marshal.StructureToPtr` and you'll see the "proper" sequential layout in the unmanaged structure. – Luaan Feb 22 '17 at 13:10
@Luaan Very interesting comment. You are right `Marshal.StructureToPtr` is "better" here. The reason why I took a pointer to a managed layout, I think, was because that was what the asker did in his question (`&test` etc.). Maybe I will add your comment to my own answer elsewhere on this page. – Jeppe Stig Nielsen Feb 22 '17 at 15:50
@luaan, except for `ExplicitLayout`, of which the msdn docs clearly say it applies to managed and unmanaged code, blittable or not – Abel Jun 02 '20 at 16:04
@Abel Nope https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.layoutkind?view=netcore-3.1#System_Runtime_InteropServices_LayoutKind_Explicit. CLR spec says the same thing. It is only guaranteed to change the unmanaged layout of the structure. It _may_ change the managed layout as well, but it's not part of the contract - you shouldn't rely on it. – Luaan Jun 03 '20 at 10:53
@luaan, the text here is explicit though: _"Use the attribute with LayoutKind.Explicit to control the precise position of each data member. This affects both managed and unmanaged layout, for both blittable and non-blittable types."_ https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.structlayoutattribute?view=netcore-3.1. And I've used that in managed code for overlapping fields, so I know it works. – Abel Jun 03 '20 at 11:17
@Abel I also know it works. That doesn't mean it's part of the contract. Overlapping fields are not supported either. That doesn't mean they don't work; it just means you can't rely on them working. – Luaan Jun 03 '20 at 12:35
@Luaan, checking ECMA-335, II.22.8, says _"The ClassLayout table is used to define how the fields of a class or value type shall be laid out by the CLI. (Normally, the CLI is free to reorder and/or insert gaps between the fields defined for a class or value type.)"_. Note the "shall" in this sentence. It mostly matches the docs (but other refs back up your claim). I doubt current behavior will ever be changed as that requires a spec change. The precise rules are in the spec as quoted and in the following sections. It also explains why `Auto` breaks layout-explicitness. – Abel Jun 03 '20 at 13:23

score 3 · Answer 4 · answered Nov 09 '10 at 14:28

3

To answer my own questions (as advised):

Question: "Does this behaviour have any ramifications when doing interop with C/C++ structs that use the Com DATETIME type?"

Answer: No, because the layout is respected when using Marshalling. (I verified this empirically.)

Question "Can anyone provide an explanation?".

Answer: I'm still not sure about this, but since the internal representation of a struct is not defined, the compiler can do what it likes.

answered Nov 09 '10 at 14:28

Matthew Watson

104,400
10
158
276

With the exception of `ExplicitLayout`, which applies to managed and unmanaged, blittable and non-blittable types. – Abel Jun 02 '20 at 14:40

score 2 · Answer 5 · answered Nov 08 '11 at 20:57

You're checking the addresses as they are within the managed structure. Marshal attributes have no guarantees for the arrangement of fields within managed structures.

The reason it marshals correctly into native structures, is because the data is copied into native memory using the attributes set by marshal values.

So, the arrangement of the managed structure has no impact on the arranged of the native structure. Only the attributes affect the arrangement of native structure.

If fields setup with marshal attributes were stored in managed data the same way as native data, then there would be no point in Marshal.StructureToPtr, you'd simply byte-copy the data over.

Note that the compiler is free to change the managed layout to be the same as the marshalled structure - it's a valid performance optimisation that allows the marshaller to avoid copying the structure in some interop scenarios. It's *not* a contractual behaviour, though - you should never make any assumptions about the managed layout of structures, regardless of `StructLayout`, `FieldOffset` etc. — Luaan, Feb 22 '17 at 13:11

erict · Answer 6 · 2011-02-14T21:24:34.207

If you're going to interop with C/C++, I would always be specific with the StructLayout. Instead of Sequential, I would go with Explicit, and specify each position with FieldOffset. In addition, add your Pack variable.

[StructLayout(LayoutKind.Explicit, Pack=1, CharSet=CharSet.Unicode)]
public struct Inner
{
    [FieldOffset(0)]
    public byte First;
    [FieldOffset(1)]
    public double NotFirst;
    [FieldOffset(9)]
    public DateTime WTF;
}

It sounds like DateTime can't be Marshaled anyhow, only to a string (bingle Marshal DateTime).

The Pack variable is especially important in C++ code that might be compiled on different systems that have different word sizes.

I would also ignore the addresses that can be seen when using unsafe code. It doesn't really matter what the compiler does as long as the Marshaling is correct.

Why does LayoutKind.Sequential work differently if a struct contains a DateTime field?

6 Answers6

Linked