Marshalling a big-endian byte collection into a struct in order to pull out values

Question

There is an insightful question about reading a C/C++ data structure in C# from a byte array, but I cannot get the code to work for my collection of big-endian (network byte order) bytes. (EDIT: Note that my real struct has more than just one field.) Is there a way to marshal the bytes into a big-endian version of the structure and then pull out the values in the endianness of the framework (that of the host, which is usually little-endian)?

(Note, reversing the array of bytes will not work - each value's bytes must be reversed, which does not give you the same collection as reversing all of the bytes.)

This should summarize what I'm looking for (LE=LittleEndian, BE=BigEndian):

void Main()
{
    var leBytes = new byte[] {1, 0, 2, 0};
    var beBytes = new byte[] {0, 1, 0, 2};
    Foo fooLe = ByteArrayToStructure<Foo>(leBytes);
    Foo fooBe = ByteArrayToStructureBigEndian<Foo>(beBytes);
    Assert.AreEqual(fooLe, fooBe);
}

[StructLayout(LayoutKind.Explicit, Size=4)]
public struct Foo  {
    [FieldOffset(0)] 
    public ushort firstUshort;
    [FieldOffset(2)] 
    public ushort secondUshort;
}

T ByteArrayToStructure<T>(byte[] bytes) where T: struct 
{
    GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
    T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(),typeof(T));
    handle.Free();
    return stuff;
}

T ByteArrayToStructureBigEndian<T>(byte[] bytes) where T: struct 
{
    ???
}

Other helpful links:

Byte of a struct and onto endian concerns

A little more on bytes and endianness (byte order)

Read binary files more efficiently using C#

Unsafe and reading from files

Mono's contribution to the issue

Mastering C# structs

Have a look at this: http://stackoverflow.com/a/2624377/1254743 It does it even more fine-grained, which you can easily change if necessary. Andyou don't need to build up your structs twice (especially nice if you have nested structs). — Onur, Feb 22 '13 at 07:58
I think the library PODCaster (http://www.zer7.com/software/podcaster and on NuGet) might be aimed at this problem, but I honestly can't tell how it's supposed to be used, even from the samples. — Pat, Nov 23 '13 at 01:02

score 14 · Answer 1 · edited May 23 '17 at 12:01

Here's another solution for swapping endianness.

It's adjusted from Adam Robinsons solution here: https://stackoverflow.com/a/2624377/1254743

It's even capable of handling nested structs.

public static class FooTest
{
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public struct Foo2
    {
        public byte b1;
        public short s;
        public ushort S;
        public int i;
        public uint I;
        public long l;
        public ulong L;
        public float f;
        public double d;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
        public string MyString;
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public struct Foo
    {
        public byte b1;
        public short s;
        public ushort S;
        public int i;
        public uint I;
        public long l;
        public ulong L;
        public float f;
        public double d;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
        public string MyString;
        public Foo2 foo2;
    }

    public static void test()
    {
        Foo2 sample2 = new Foo2()
        {
            b1 = 0x01,
            s = 0x0203,
            S = 0x0405,
            i = 0x06070809,
            I = 0x0a0b0c0d,
            l = 0xe0f101112131415,
            L = 0x161718191a1b1c,
            f = 1.234f,
            d = 4.56789,
            MyString = @"123456789", // null terminated => only 9 characters!
        };

        Foo sample = new Foo()
        {
            b1 = 0x01,
            s = 0x0203,
            S = 0x0405,
            i = 0x06070809,
            I = 0x0a0b0c0d,
            l = 0xe0f101112131415,
            L = 0x161718191a1b1c,
            f = 1.234f,
            d = 4.56789,
            MyString = @"123456789", // null terminated => only 9 characters!
            foo2 = sample2,
        };

        var bytes_LE = Dummy.StructToBytes(sample, Endianness.LittleEndian);
        var restoredLEAsLE = Dummy.BytesToStruct<Foo>(bytes_LE, Endianness.LittleEndian);
        var restoredLEAsBE = Dummy.BytesToStruct<Foo>(bytes_LE, Endianness.BigEndian);

        var bytes_BE = Dummy.StructToBytes(sample, Endianness.BigEndian);
        var restoredBEAsLE = Dummy.BytesToStruct<Foo>(bytes_BE, Endianness.LittleEndian);
        var restoredBEAsBE = Dummy.BytesToStruct<Foo>(bytes_BE, Endianness.BigEndian);

        Debug.Assert(sample.Equals(restoredLEAsLE));
        Debug.Assert(sample.Equals(restoredBEAsBE));
        Debug.Assert(restoredBEAsLE.Equals(restoredLEAsBE));
    }

    public enum Endianness
    {
        BigEndian,
        LittleEndian
    }

    private static void MaybeAdjustEndianness(Type type, byte[] data, Endianness endianness, int startOffset = 0)
    {
        if ((BitConverter.IsLittleEndian) == (endianness == Endianness.LittleEndian))
        {
            // nothing to change => return
            return;
        }

        foreach (var field in type.GetFields())
        {
            var fieldType = field.FieldType;
            if (field.IsStatic)
                // don't process static fields
                continue;

            if (fieldType == typeof(string)) 
                // don't swap bytes for strings
                continue;

            var offset = Marshal.OffsetOf(type, field.Name).ToInt32();

            // handle enums
            if (fieldType.IsEnum)
                fieldType = Enum.GetUnderlyingType(fieldType);

            // check for sub-fields to recurse if necessary
            var subFields = fieldType.GetFields().Where(subField => subField.IsStatic == false).ToArray();

            var effectiveOffset = startOffset + offset;

            if (subFields.Length == 0)
            {
                Array.Reverse(data, effectiveOffset, Marshal.SizeOf(fieldType));
            }
            else
            {
                // recurse
                MaybeAdjustEndianness(fieldType, data, endianness, effectiveOffset);
            }
        }
    }

    internal static T BytesToStruct<T>(byte[] rawData, Endianness endianness) where T : struct
    {
        T result = default(T);

        MaybeAdjustEndianness(typeof(T), rawData, endianness);

        GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);

        try
        {
            IntPtr rawDataPtr = handle.AddrOfPinnedObject();
            result = (T)Marshal.PtrToStructure(rawDataPtr, typeof(T));
        }
        finally
        {
            handle.Free();
        }

        return result;
    }

    internal static byte[] StructToBytes<T>(T data, Endianness endianness) where T : struct
    {
        byte[] rawData = new byte[Marshal.SizeOf(data)];
        GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);
        try
        {
            IntPtr rawDataPtr = handle.AddrOfPinnedObject();
            Marshal.StructureToPtr(data, rawDataPtr, false);
        }
        finally
        {
            handle.Free();
        }

        MaybeAdjustEndianness(typeof(T), rawData, endianness);

        return rawData;
    }

}

I suspect this code will end up reversing the order of any elements inside of ByValArrays... a special case will need to be added, like one was added for enums, to operate on arrays if desired. — user169771, Jun 24 '21 at 05:52
Works for me! [StructLayout(LayoutKind.Sequential, Pack = 1)] — Latency, Aug 17 '21 at 01:49

Pat · Accepted Answer · 2011-05-27T21:12:12.417

As alluded to in my comment on @weismat's answer, there is an easy way to achieve big-endian structuring. It involves a double-reversal: the original bytes are reversed entirely, then the struct itself is the reversal of the original (big-endian) data format.

The fooLe and fooBe in Main will have the same values for all fields. (Normally, the little-endian struct and bytes wouldn't be present, of course, but this clearly shows the relationship between the byte orders.)

NOTE: See updated code including how to get bytes back out of the struct.

public void Main()
{
    var beBytes = new byte[] {
        0x80, 
        0x80,0, 
        0x80,0, 
        0x80,0,0,0, 
        0x80,0,0,0,
        0x80,0,0,0,0,0,0,0, 
        0x80,0,0,0,0,0,0,0, 
        0x3F,0X80,0,0, // float of 1 (see http://en.wikipedia.org/wiki/Endianness#Floating-point_and_endianness)
        0x3F,0xF0,0,0,0,0,0,0, // double of 1
        0,0,0,0x67,0x6E,0x69,0x74,0x73,0x65,0x54 // Testing\0\0\0
    };
    var leBytes = new byte[] {
        0x80, 
        0,0x80,
        0,0x80, 
        0,0,0,0x80,
        0,0,0,0x80, 
        0,0,0,0,0,0,0,0x80, 
        0,0,0,0,0,0,0,0x80, 
        0,0,0x80,0x3F, // float of 1
        0,0,0,0,0,0,0xF0,0x3F, // double of 1
        0x54,0x65,0x73,0x74,0x69,0x6E,0x67,0,0,0 // Testing\0\0\0
    };
    Foo fooLe = ByteArrayToStructure<Foo>(leBytes).Dump("LE");
    FooReversed fooBe = ByteArrayToStructure<FooReversed>(beBytes.Reverse().ToArray()).Dump("BE");  
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Foo  {
    public byte b1;
    public short s;
    public ushort S;
    public int i;
    public uint I;
    public long l;
    public ulong L;
    public float f;
    public double d;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string MyString;
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct FooReversed  {
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string MyString;
    public double d;
    public float f;
    public ulong L;
    public long l;
    public uint I;
    public int i;
    public ushort S;
    public short s;
    public byte b1;
}

T ByteArrayToStructure<T>(byte[] bytes) where T: struct 
{
    GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
    T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(),typeof(T));
    handle.Free();
    return stuff;
}

score 1 · Answer 3 · edited Jan 18 '13 at 16:05

I agree with @weismat and believe there is no solution.

What you show in your example is that you can access a raw byte buffer as if it were any OTHER structure without changing anything to it, not copying or moving data around, nothing. Just pinning it to avoid it to move around because of GC.

This is basically what you usually achieve in C by using a union type containing both your target structure and a byte array of the same size.

The good side is that it is really efficient.

That has several drawbacks, the main one being that you can only get access this way to data that are in the native machine order (be it LE or BE). Hence your ByteArrayToStructure is not really LE, it is only so because the processor underneath is LE. If you compile the same program on another target that happen to be BE, it works the other way and believe your byte array is BE.

Other drawbacks are that you must be very cautious with data alignment, be aware of possible padding, etc. and of course that there is no way to change byte order from LE to BE without moving data in bytes array (if you have a 16 bits integers only array as in your example this is merely swapping every two bytes).

I happened to have a similar problem and poundered not to use this solution because of the previous drawbacks and opted to hide my input structures behind accessors to hide access to the bytes array underneath. It may not be as elegant, but it is simple and also avoid to copy the buffer or move data in any way.

score 1 · Answer 4 · answered Mar 30 '10 at 22:02

It seems there must be a more elegant solution, but this should at least get you going:

    static T ByteArrayToStructureBigEndian<T>(byte[] bytes) where T : struct
    {
        GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
        T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
        handle.Free();
        System.Type t = stuff.GetType();
        FieldInfo[] fieldInfo = t.GetFields();
        foreach (FieldInfo fi in fieldInfo)
        {                 
            if (fi.FieldType == typeof(System.Int16))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.Int32))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.Int64))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.UInt16))
            {
                UInt16 i16 = (UInt16)fi.GetValue(stuff);
                byte[] b16 = BitConverter.GetBytes(i16);
                byte[] b16r = b16.Reverse().ToArray();
                fi.SetValueDirect(__makeref(stuff), BitConverter.ToUInt16(b16r, 0);
            }
            else if (fi.FieldType == typeof(System.UInt32))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.UInt64))
            {
                // TODO
            }
        }
        return stuff;
    }

Ah yes, reflection - I was afraid that someone would answer with reflection ;-). — Pat, Mar 31 '10 at 20:30
SetValueDirect requires a TypedReference (a combination of a managed pointer and the type of what is pointed to) to the data structure who's member we're setting. The __makeref function returns this. — 500 - Internal Server Error, Jan 31 '11 at 17:47
Worse performance than sending an email to a mail server on the moon and then going there via a benz patent motor car to get the contents. 99% of the time serialization needs to be performant, don't use this unless you are on the 1%. — Orestis P., Oct 07 '18 at 11:50

Pat · Answer 5 · 2010-06-17T20:10:06.777

I finally figured out a way that didn't involve reflection and is mostly user-friendly. It uses Mono's DataConverter class (source) which, unfortunately, is fairly buggy at this point. (For example, floats and doubles don't seem to work correctly, string parsing is broken, etc.)

The trick is to Unpack and re-pack the bytes as big-endian, which requires a string describing what types are in the byte array (see the last method).Also, byte alignment is tricky: there are four bytes in the struct instead of one because marshaling seems to rely on having 4-byte alignment (I still don't quite understand that part). (EDIT: I have found that adding Pack=1 to the StructLayout attribute usually takes care of byte-alignment issues.)

Note, this sample code was used in LINQPad - the Dump extension method just prints info about the object and returns the object (it's fluent).

public void Main()
{
    var beBytes = new byte[] {
        0x80, 
        0x80, 
        0x80, 
        0x80, 
        0x80,0, 
        0x80,0, 
        0x80,0,0,0, 
        0x80,0,0,0,
        0x80,0,0,0,0,0,0,0, 
        0x80,0,0,0,0,0,0,0, 
//      0,0,0x80,0x3F, // float of 1
//      0,0,0,0,0,0,0xF0,0x3F, // double of 1
        0x54,0x65,0x73,0x74,0x69,0x6E,0x67,0,0,0 // Testing\0\0\0
    };
    var leBytes = new byte[] {
        0x80, 
        0x80, 
        0x80, 
        0x80, 
        0,0x80,
        0,0x80, 
        0,0,0,0x80,
        0,0,0,0x80, 
        0,0,0,0,0,0,0,0x80, 
        0,0,0,0,0,0,0,0x80, 
//      0,0,0x80,0x3F, // float of 1
//      0,0,0,0,0,0,0xF0,0x3F, // double of 1
        0x54,0x65,0x73,0x74,0x69,0x6E,0x67,0,0,0 // Testing\0\0\0
    };
    Foo fooLe = ByteArrayToStructure<Foo>(leBytes).Dump("LE");
    Foo fooBe = ByteArrayToStructureBigEndian<Foo>(beBytes, 
        "bbbbsSiIlL"
//      + "fd" // float, then double
        +"9bb").Dump("BE");
    Assert.AreEqual(fooLe, fooBe);
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Foo  {
    public byte b1;
    public byte b2;
    public byte b3;
    public byte b4;
    public short s;
    public ushort S;
    public int i;
    public uint I;
    public long l;
    public ulong L;
//  public float f;
//  public double d;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string MyString;
}

T ByteArrayToStructure<T>(byte[] bytes) where T: struct 
{
    GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
    T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(),typeof(T));
    handle.Free();
    return stuff;
}

T ByteArrayToStructureBigEndian<T>(byte[] bytes, string description) where T: struct 
{
    byte[] buffer = bytes;
    IList unpacked = DataConverter.Unpack("^"+description, buffer, 0).Dump("unpacked");
    buffer = DataConverter.PackEnumerable("!"+description, unpacked).Dump("packed");
    return ByteArrayToStructure<T>(buffer);
}

score 0 · Answer 6 · answered Mar 19 '10 at 19:54

0

Have you tried MiscUtil? It has a utility class named EndianBitConverter to convert between big and little endian byte arrays.

http://www.yoda.arachsys.com/csharp/miscutil/

answered Mar 19 '10 at 19:54

JaredPar

733,204
149
1,241
1,454

2

Yes. That won't work because it deals with only the bytes necessary for a certain value, e.g. turning `{0, 1}` into a `ushort` of `1`, not for entire structures. – Pat Mar 19 '10 at 20:17

score -1 · Answer 7 · answered Mar 19 '10 at 19:59

-1

From my point of view you just need to add an Array.Reverse() before the conversion of the byte array.

answered Mar 19 '10 at 19:59

weismat

7,195
3
43
58

1

See updates to my question that should clarify (even better) that reversing the array does not work because there are multiple values in the struct. – Pat Mar 19 '10 at 20:18
Ok - but I doubt then that there is a generic solution possible as you need to know the field sizes for reversing - you need to reverse then the GetBytes from the BitConverter class per field. – weismat Mar 19 '10 at 20:26
Right, which is what I am currently doing. But the solution for the little-endian case is so elegant, I want it to work for my big-endian case! – Pat Mar 19 '10 at 20:34
Well, I obviously wasn't thinking this through all of the way. If the bytes are entirely reversed, the struct can take the format of the data in the reverse order then each field is correctly reversed! – Pat May 18 '11 at 21:12

score -1 · Answer 8 · answered Mar 31 '10 at 03:03

The traditional solution is to use ntohl() and ntohs().

typedef struct {
  long foo;
  short bar, baz;
  char xyzzy;
} Data;

Data d;
memcpy(&d, buffer, sizeof(Data));

d.foo = ntohl(d.foo);
d.bar = ntohs(d.bar);
d.baz = ntohs(d.baz);
// don't need to change d.xyxxy

The above works on any platform that has BSD Sockets, no matter whether it's big-endian, little-endian, or something utterly weird like a VAX. The reverse operation is done using hton*().

On big-endian platforms the functions are usually no-ops and should therefore be zero cost.

Marshalling a big-endian byte collection into a struct in order to pull out values

8 Answers8

Linked

Related