Why raw disk read in C# reads from slightly shifted offset?

Question

I'm trying to read raw disk. I've successfully opened the drive and obtained valid handle (CreateFile), set offset for that handle to zero (SetFilePointerEx) and read the data to a buffer Byte[] (ReadFile). So far so good. But for some unknown reason, when I compare the buffer (took my a while to figure it out actually) to what 3rd party utilities (Disk Investigator) shows. It doesn't contain a boot information (jump instruction), but correct disk data, but starting at offset 85 = 0x55. So somewhere in the middle of volume boot information.

Why is that? Is there something, I'm missing (obviously I am)?

System: Windows 8 (in VmWare Workstation)

Code:

// moves the pointer to a given offset
Int64 offset = 0; 
Int64 newOffset;
Int32 bufferSize = (Int32) clusterSizeInBytes; 
SetFilePointerEx(driveHandle.Handle, offset, out newOffset, WinMoveMethod.Begin);
Int32 error = Marshal.GetLastWin32Error();

// reads the raw buffer
Int32 numberOfBytesRead;
rawData = new Byte[bufferSize];
Boolean result = ReadFile(driveHandle.Handle, rawData, bufferSize, 
                          out numberOfBytesRead, IntPtr.Zero);

CreateFile (elsewhere)

driveHandle = CreateFile("\\.\PhysicalDrive1", WinFileAccessMode.GenericRead, 
           WinFileSharedAccess.All, IntPtr.Zero, WinFileMode.OpenExisting,
           WinFileAttribute.None, IntPtr.Zero);

PInvoke methods:

[DllImport("kernel32.dll", SetLastError = true)]
public static extern Boolean SetFilePointerEx(
    [In] SafeFileHandle fileHandle, 
    [In] Int64 distanceToMove,
    [Out] out Int64 newOffset, 
    [In] WinMoveMethod moveMethod);

[DllImport("kernel32", SetLastError = true)]
public extern static Boolean ReadFile(
    [In] SafeFileHandle handle, 
    [Out] Byte[] buffer,
    [In] Int32 numBytesToRead, 
    [Out] out Int32 numBytesRead, 
    [In] IntPtr overlapped);

Enumerations:

public enum WinMoveMethod : uint
{   
    Begin   = 0,
    Current = 1,
    End     = 2
}

I did not get the part about boot information. What did you try to achieve? Does your read show different data than the disk util *at the same offset*? Maybe you opened a partition in one and a volume in the other? — usr, Mar 03 '13 at 12:04
I'm just trying to read raw disk. It should start with NTFS Boot Sector's jump instruction (0xEB5290 at offset 0). But when I read handle from \\.\PhysicalDrive2 it just reads from offset 85. So I can't read the first 84 bytes. Even though SetFilePointerEx is set to 0. — SmartK8, Mar 03 '13 at 12:20
I find it hard to believe that it reads from an uneven offset because that would shift the entire disk. You would be unable to read full disk sectors (of say 512 bytes) because of the offset. That can't be true. Your bug is elsewhere. Probably you're misinterpreting the data. Can you show what you got and what you expected? — usr, Mar 03 '13 at 12:29
OK. I'm puzzled as you are. When I use non-aligned read it fails as one would expect, when I read from offset < 0 it fails as well (as expected). SetFilePointerEx returns correctly 0 (no errors). It's a puzzle. My guess would be a PInvoke definition of readfile or something with out buffer. I'll add some code and definitions. — SmartK8, Mar 03 '13 at 12:32
I've added some more code. Sorry, it is a part of large project, so it's hard to separate the bits (especially enumerations). — SmartK8, Mar 03 '13 at 12:46
I've now found out. That it's probably worse than that. I've checked the sequence and it doesn't match. 33 C0 8E D0 BC 00 7C matches the offset 85 in Disk Inspector but after that it is different (for the same volume/drive). Unbelievable. So the buffer may be potentially completely wrong. What a coincidental match. I now don't know where I stand. But both SetFilePointerEx and ReadFile returns correct result, and number of bytes read at least. As for the buffer data, I'm not sure anymore. — SmartK8, Mar 03 '13 at 13:14
Maybe the data is from some even later offset (like from the 1. partition)? Scan the entire disk for a longer sequence. — usr, Mar 03 '13 at 14:11

metadings · Accepted Answer · 2013-03-03T13:28:18.103

Can't find anymore a library using a namespace "DiskLib". I can't test that or completely upload now, even I don't know anymore who wrote that piece of code, but the lines you may find interesting are

public class DiskStream : Stream
{
    public const int DEFAULT_SECTOR_SIZE = 512;
    private const int BUFFER_SIZE = 4096;

    private string diskID;
    private DiskInfo diskInfo;
    private FileAccess desiredAccess;
    private SafeFileHandle fileHandle;

    public DiskInfo DiskInfo
    {
        get { return this.diskInfo; }
    }
    public uint SectorSize
    {
        get { return this.diskInfo.BytesPerSector; }
    }

    public DiskStream(string diskID, FileAccess desiredAccess)
    {
        this.diskID = diskID;
        this.diskInfo = new DiskInfo(diskID);
        this.desiredAccess = desiredAccess;

        // if desiredAccess is Write or Read/Write
        //   find volumes on this disk
        //   lock the volumes using FSCTL_LOCK_VOLUME
        //     unlock the volumes on Close() or in destructor


        this.fileHandle = this.openFile(diskID, desiredAccess);
    }

    private SafeFileHandle openFile(string id, FileAccess desiredAccess)
    {
        uint access;
        switch (desiredAccess)
        {
            case FileAccess.Read:
                access = DeviceIO.GENERIC_READ;
                break;
            case FileAccess.Write:
                access = DeviceIO.GENERIC_WRITE;
                break;
            case FileAccess.ReadWrite:
                access = DeviceIO.GENERIC_READ | DeviceIO.GENERIC_WRITE;
                break;
            default:
                access = DeviceIO.GENERIC_READ;
                break;
        }

        SafeFileHandle ptr = DeviceIO.CreateFile(
            id,
            access,
            DeviceIO.FILE_SHARE_READ,
            IntPtr.Zero,
            DeviceIO.OPEN_EXISTING,
            DeviceIO.FILE_FLAG_NO_BUFFERING | DeviceIO.FILE_FLAG_WRITE_THROUGH,
            IntPtr.Zero);

        if (ptr.IsInvalid)
        {
            Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }

        return ptr;
    }

    public override bool CanRead
    {
        get
        {
            return (this.desiredAccess == FileAccess.Read || this.desiredAccess == FileAccess.ReadWrite) ? true : false;
        }
    }
    public override bool CanWrite
    {
        get
        {
            return (this.desiredAccess == FileAccess.Write || this.desiredAccess == FileAccess.ReadWrite) ? true : false;
        }
    }
    public override bool CanSeek
    {
        get
        {
            return true;
        }
    }
    public override long Length {
      get { return (long)this.Length; }
    }

    public ulong LengthU
    {
        get { return this.diskInfo.Size; }
    }

    public override long Position {
      get {
        return (long)PositionU;
      }
      set {
        PositionU = (ulong)value;
      }
    }

    public ulong PositionU
    {
        get
        {
            ulong n = 0;
            if (!DeviceIO.SetFilePointerEx(this.fileHandle, 0, out n, (uint)SeekOrigin.Current))
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            return n;
        }
        set
        {
            if (value > (this.LengthU - 1))
                throw new EndOfStreamException("Cannot set position beyond the end of the disk.");

            ulong n = 0;
            if (!DeviceIO.SetFilePointerEx(this.fileHandle, value, out n, (uint)SeekOrigin.Begin))
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }
    }

    public override void Flush()
    {
        // not required, since FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING are used
        //if (!Unmanaged.FlushFileBuffers(this.fileHandle))
        //    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
    }
    public override void Close()
    {
      if (this.fileHandle != null) {
        DeviceIO.CloseHandle(this.fileHandle);
        this.fileHandle.SetHandleAsInvalid();
        this.fileHandle = null;
      }
      base.Close();
    }

    public override void SetLength(long value)
    {
        throw new NotSupportedException("Setting the length is not supported with DiskStream objects.");
    }
    public override int Read(byte[] buffer, int offset, int count) {
      return (int)Read(buffer, (uint)offset, (uint)count);
    }

    public unsafe uint Read(byte[] buffer, uint offset, uint count)
    {
        uint n = 0;
        fixed (byte* p = buffer)
        {
            if (!DeviceIO.ReadFile(this.fileHandle, p + offset, count, &n, IntPtr.Zero))
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }
        return n;
    }
    public override void Write(byte[] buffer, int offset, int count) {
      Write(buffer, (uint)offset, (uint)count);
    }
    public unsafe void Write(byte[] buffer, uint offset, uint count)
    {
        uint n = 0;
        fixed (byte* p = buffer)
        {
            if (!DeviceIO.WriteFile(this.fileHandle, p + offset, count, &n, IntPtr.Zero))
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }
    }
    public override long Seek(long offset, SeekOrigin origin) {
      return (long)SeekU((ulong)offset, origin);
    }
    public ulong SeekU(ulong offset, SeekOrigin origin)
    {
        ulong n = 0;
        if (!DeviceIO.SetFilePointerEx(this.fileHandle, offset, out n, (uint)origin))
            Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        return n;
    }

    public uint ReadSector(DiskSector sector)
    {
        return this.Read(sector.Data, 0, sector.SectorSize);
    }
    public void WriteSector(DiskSector sector)
    {
        this.Write(sector.Data, 0, sector.SectorSize);
    }
    public void SeekSector(DiskSector sector)
    {
        this.Seek(sector.Offset, SeekOrigin.Begin);
    }
}

The DeviceIO class is a p/invoke boilerplate for Win32 API.

The DiskInfo class is a wrapper to WMI Win32_DiskDrive and Win32_DiskPartition classes.

I used that library once to clone a disk (on Win7). Hope that helps to find the solution.

For completeness and because of the details, the DeviceIO class is throughout using unsigned integers:

/// <summary>
/// P/Invoke wrappers around Win32 functions and constants.
/// </summary>
internal partial class DeviceIO
{

    #region Constants used in unmanaged functions

    public const uint FILE_SHARE_READ = 0x00000001;
    public const uint FILE_SHARE_WRITE = 0x00000002;
    public const uint FILE_SHARE_DELETE = 0x00000004;
    public const uint OPEN_EXISTING = 3;

    public const uint GENERIC_READ = (0x80000000);
    public const uint GENERIC_WRITE = (0x40000000);

    public const uint FILE_FLAG_NO_BUFFERING = 0x20000000;
    public const uint FILE_FLAG_WRITE_THROUGH = 0x80000000;
    public const uint FILE_READ_ATTRIBUTES = (0x0080);
    public const uint FILE_WRITE_ATTRIBUTES = 0x0100;
    public const uint ERROR_INSUFFICIENT_BUFFER = 122;

    #endregion

    #region Unamanged function declarations

    [DllImport("kernel32.dll", SetLastError = true)]
    public static unsafe extern SafeFileHandle CreateFile(
        string FileName,
        uint DesiredAccess,
        uint ShareMode,
        IntPtr SecurityAttributes,
        uint CreationDisposition,
        uint FlagsAndAttributes,
        IntPtr hTemplateFile);

    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool CloseHandle(SafeFileHandle hHandle);

    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool DeviceIoControl(
        SafeFileHandle hDevice,
        uint dwIoControlCode,
        IntPtr lpInBuffer,
        uint nInBufferSize,
        [Out] IntPtr lpOutBuffer,
        uint nOutBufferSize,
        ref uint lpBytesReturned,
        IntPtr lpOverlapped);

    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern unsafe bool WriteFile(
        SafeFileHandle hFile,
        byte* pBuffer,
        uint NumberOfBytesToWrite,
        uint* pNumberOfBytesWritten,
        IntPtr Overlapped);

    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern unsafe bool ReadFile(
        SafeFileHandle hFile,
        byte* pBuffer,
        uint NumberOfBytesToRead,
        uint* pNumberOfBytesRead,
        IntPtr Overlapped);

    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool SetFilePointerEx(
        SafeFileHandle hFile,
        ulong liDistanceToMove,
        out ulong lpNewFilePointer,
        uint dwMoveMethod);

    [DllImport("kernel32.dll")]
    public static extern bool FlushFileBuffers(
        SafeFileHandle hFile);

    #endregion

}

Thanks. I already know how to do it, but it reads from offset 85 instead of 0. So I'm missing 84 bytes. That's the core of the problem. — SmartK8, Mar 03 '13 at 12:26
@SmartK8 sounds to be a detail. do you use offsets and lengths being multiples of BytesPerSector? — metadings, Mar 03 '13 at 12:41
Yes, I do. But the data were weird, so I've tried to set offset to 0 (to be sure). And expected to see jump instruction of boot sector. What I've got was 33C0BE.. instead. So I've used Disk Inspector to search for it. And it was found at raw disk offset 85. Even though I'm setting offset to 0, and it succeeds. — SmartK8, Mar 03 '13 at 12:45
@SmartK8 beware that you read (the MBR) forwards but you need to parse it backwards (it's little endian)... — metadings, Mar 03 '13 at 12:51
I know that, but unfortunately, I wasn't able to get to that, because ReadFile just doesn't return correct buffer. Probably some definition mismatch. But it won't fail nonetheless. — SmartK8, Mar 03 '13 at 13:16
@SmartK8 Use unsigned integers! Change all Int32 to UInt32 and all Int64 (long) to UInt64 (ulong), also in the p/invoke definitions! — metadings, Mar 03 '13 at 13:23
OK, I'll try that. Edit: Nothing, same results. That Byte[] is all zeros, and after ReadFile it is filled up with consistent data (they're same each time). So I'm definitely reading something. I just don't know what, and what offset. — SmartK8, Mar 03 '13 at 13:24
OK, I figured it out. I've installed WinHex. I first opened a volume handle. The situation was same as with Disk Inspector. Then I've noticed that there's physical media (drive). I've opened that one I did, and voilà the same data as mine. So I'm probably reading the drive handle, but I should be reading volume handle. Which now seems logical. I going to try the volume handle instead. And I'll get back to you. — SmartK8, Mar 03 '13 at 13:34
That's it.. (non)problem solved. I'll accept your answer as it is basically correct. Problem was the type of input handle. :) — SmartK8, Mar 03 '13 at 13:42
you remind me banging my head on the wall, all nights long. but it's pretty cool stuff. thanks ;) — metadings, Mar 03 '13 at 13:43
@SmartK8 sorry on the old post, but how did you offset to the volume handle as I am having a similar issue and need the logical handle? — vipersassassin, Jul 12 '17 at 20:06
As I've written in comments I was using disk (drive) handle instead of volume handle. Thus it read bytes from the start of the drive instead of start of the given volume. I just used volume handle i.e. C,D, etc. instead of disk (drive) 0,1, etc. It was offset automatically from a volume handle. — SmartK8, Jul 14 '17 at 08:11

Why raw disk read in C# reads from slightly shifted offset?

1 Answers1