Get type of binary on filesystem via C# running in 64 bit

Question

I have a C# application, compiled in Visual Studio 2017 on the 'Any CPU' target, with the 'Prefer 32-bit' option disabled. In this application, I am trying to pinvoke kernel32!GetBinaryType(). When running with 'Prefer 32-bit' enabled, it works fine. When running in either 32 or 64-bit mode from a C++ executable, it works fine. I am not sure what I am doing wrong with the 64-bit C# application.

This is my pinvoke signature:

[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool GetBinaryTypeW([MarshalAs(UnmanagedType.LPWStr)] string path, out UInt32 result);

Calling this from 64 bit mode, GetLastError() returns 193, which according to https://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx is

ERROR_BAD_EXE_FORMAT.

I have attached a 64-bit debugger to both my C# application and my C++ application to make sure that the string is reaching GetBinaryTypeW() in the correct place on the stack, and it appears to me that it is.

I am concerned that I have run into some subtle bug in the marshaler.

What is wrong with my approach?

Update

The question referenced in the comments does not match this situation. In that question, a call to LoadLibrary() failed because it was being used to try and load a 32 bit DLL into a 64 bit process. I am not trying to load any DLL. I am merely trying to use GetBinaryType() to examine the PE header of an executable.

Answer 1

GetBinaryType does some funny things when the process is running in the WOW6432 space, and the .NET runtime may exacerbate this. Unfortunately, I don't recall all the details right now. However, here's a solution which is more robust and which works for both EXEs and DLLs. Note that this only detects the PE binary type. If you are dealing with .NET assemblies that are compiled as AnyCPU, you may or may not get the result you expect. I will leave any further investigation as an exercise for the reader.

public enum BinaryType : uint
{
    SCS_32BIT_BINARY = 0,
    SCS_64BIT_BINARY = 6,
    SCS_DOS_BINARY = 1,
    SCS_OS216_BINARY = 5,
    SCS_PIF_BINARY = 3,
    SCS_POSIX_BINARY = 4,
    SCS_WOW_BINARY = 2
}

public static BinaryType? GetBinaryType(string path)
{
    using (FileStream stream = new FileStream(path, FileMode.Open, FileAccess.Read))
    {
        stream.Seek(0x3C, SeekOrigin.Begin);
        using (var reader = new BinaryReader(stream))
        {
            if (stream.Position + sizeof(int) > stream.Length)
                return null;
            var peOffset = reader.ReadInt32();
            stream.Seek(peOffset, SeekOrigin.Begin);
            if (stream.Position + sizeof(uint) > stream.Length)
                return null;
            var peHead = reader.ReadUInt32();
            if (peHead != 0x00004550) // "PE\0\0"
                return null;
            if (stream.Position + sizeof(ushort) > stream.Length)
                return null;
            switch (reader.ReadUInt16())
            {
                case 0x14c:
                    return BinaryType.SCS_32BIT_BINARY;
                case 0x8664:
                    return BinaryType.SCS_64BIT_BINARY;
                default:
                    return null;
            }
        }
    }
}