RawInput WinAPI: GetRawInputBuffer() and message handling

Question

Why
I'm trying to get input from a barcode scanner to my (visual) application. I would like to ignore input from other devices and get the input even if the application loses focus. I found the RawInput API recommended on SO and also elsewhere, to achieve this.

I've focused on GetRawInputBuffer() to read the input, as I'm expecting ~2 scans per second and ~700 events (key down / key up) triggered for each scan (assuming the scanner is acting as a keyboard). The documentation mentions to use GetRawInputBuffer() "for devices that can produce large amounts of raw input". I don't know whether the above actually qualifies...

Problem
I've successfully received input data - but there is something I must be doing wrong (possibly fundamentally...) as I can't figure out a good way to get consistent results. The raw data seems to 'disappear' very quickly and I often get no data back. There are similar existing questions on SO about GetRawInputBuffer() but they have only gotten me so far... Some notes:

• I use 'unRawInput.pas' from https://github.com/lhengen/RawInput
• I create a window handle using AllocateHwnd() and register it for raw input

(edit) Question
How/when should I (correctly) call GetRawInputBuffer() in a visual application to get consistent results, meaning e.g. all key events since the last call? Or: How/why do events seem to get 'discarded' between calls and how can I prevent it?

Code
The below code is a 64bit console application showcasing 3 approaches I've tried so far, and their problems (uncomment / comment-out approaches as described in code comments of the main begin-end.-block).

• approach #1: Sleep() while input is happening, then reading the buffer right away. I got the idea to Sleep() from the learn.microsoft.com sample code - and it works very well in that it seems to get all the input, but I don't think this is practical as my application needs to remain responsive.
• approach #2: use GetMessage() - usually, this yields no data, unless you type very quickly (like, mash keys) and even then, it's maybe 50% of input, tops.
• approach #3: use PeekMessage() and PM_NOREMOVE - this seems to get input very consistently but maxes out the thread.

program readrawbuffer;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  WinAPI.Windows,
  WinAPI.Messages,
  System.Classes,
  System.SysUtils,
  URawInput in '..\URawInput.pas';       // from: https://github.com/lhengen/RawInput

type
  TGetInput = class
  strict private
    fRawInputStructureSize: UINT;
    fRawInputHeaderSize: UINT;
    fRawInputBufferSize: Cardinal;
    fRawInputDevice: RAWINPUTDEVICE;
    fRawInputBuffer: PRAWINPUT;
    procedure RawInputWndProc(var aMsg: TMessage);
  public
    fRawInputWindowHnd: HWND;
    function ReadInputBuffer(): String;
    constructor Create();
    destructor Destroy(); override;
  end;

  constructor TGetInput.Create();
  begin
    inherited;
    fRawInputStructureSize := SizeOf(RAWINPUT);
    fRawInputHeaderSize := SizeOf(RAWINPUTHEADER);
    // create buffer
    fRawInputBufferSize := 40 * 16;
    GetMem(fRawInputBuffer, fRawInputBufferSize);
    // create handle and register for raw (keyboard) input
    fRawInputWindowHnd := AllocateHWnd(RawInputWndProc);
    fRawInputDevice.usUsagePage := $1;
    fRawInputDevice.usUsage := $6;
    fRawInputDevice.dwFlags := RIDEV_INPUTSINK;
    fRawInputDevice.hwndTarget := fRawInputWindowHnd;
    if RegisterRawInputDevices(@fRawInputDevice, 1, SizeOf(RAWINPUTDEVICE)) then
      WriteLn('device(s) registered; start typing...')
    else
      WriteLn('error registering device(s): ' + GetLastError().ToString());
  end;

  destructor TGetInput.Destroy();
  begin
    if Assigned(fRawInputBuffer) then
      FreeMem(fRawInputBuffer);

    DeallocateHWnd(fRawInputWindowHnd);
    inherited;
  end;

  function TGetInput.ReadInputBuffer(): String;
  var
    pcbSize, pcbSizeT: UINT;
    numberOfStructs: UINT;
    pRI: PRAWINPUT;

  begin
    Result := String.Empty;
    pcbSize := 0;
    pcbSizeT := 0;

    numberOfStructs := GetRawInputBuffer(nil, pcbSize, fRawInputHeaderSize);
    if (numberOfStructs = 0) then
    begin
      // learn.microsoft.com says for 'nil'-call: "minimum required buffer, in bytes, is returned in *pcbSize"
      // though probably redundant, I guess it can't hurt to check:
      if (fRawInputBufferSize < pcbSize) then
      begin
        fRawInputBufferSize := pcbSize * 16;
        ReallocMem(fRawInputBuffer, fRawInputBufferSize);
      end;

      repeat
        pcbSizeT := fRawInputBufferSize;
        numberOfStructs := GetRawInputBuffer(fRawInputBuffer, pcbSizeT, fRawInputHeaderSize);
        if ((numberOfStructs > 0) and (numberOfStructs < 900000)) then
        begin
          {$POINTERMATH ON}
          pRI := fRawInputBuffer;

          for var i := 0 to (numberOfStructs - 1) do
          begin
            if (pRI.keyboard.Flags = RI_KEY_MAKE) then
              Result := Result + pRI.keyboard.VKey.ToHexString() + #32;

            pRI := NEXTRAWINPUTBLOCK(pRI);
          end;
          {$POINTERMATH OFF}
          // DefRawInputProc();   // doesn't do anything? http://blog.airesoft.co.uk/2014/04/defrawinputproc-rastinating-away/
        end
        else
          Break;
      until False;

    end
  end;

  procedure TGetInput.RawInputWndProc(var aMsg: TMessage);
  begin
    // comment-out case block for Sleep() approach; leave last DefWindowProc() line
    // leave case block for GetMessage() / PeekMessage() -approaches; comment-out last DefWindowProc() line
//    case aMsg.Msg of
//      WM_INPUT:
//        begin
//          Write(ReadInputBuffer(), '-');
//          aMsg.Result := 0;
//        end
//    else
//      aMsg.Result := DefWindowProc(fRawInputWindowHnd, aMsg.Msg, aMsg.WParam, aMsg.LParam);
//    end;

    // comment-out for GetMessage() / PeekMessage() -approaches
    aMsg.Result := DefWindowProc(fRawInputWindowHnd, aMsg.Msg, aMsg.WParam, aMsg.LParam);
  end;


var
  getInput: TGetInput;
  lpMsg: tagMSG;

begin
  getInput := TGetInput.Create();


////////////////////////////////////////////////////////////////////////////////
// approach #1: Sleep()
// >> comment-out other aproaches; comment-out case block in RawInputWndProc(), leave last DefWindowProc() line

  repeat
    WriteLn('sleeping, type now...');
    Sleep(3000);
    WriteLn('VKeys read: ', getInput.ReadInputBuffer());
  until False;


////////////////////////////////////////////////////////////////////////////////
// approach #2: GetMessage()
// >> comment-out other approaches; comment-out last DefWindowProc() line in RawInputWndProc(), leave case block

//  repeat
//    // learn.microsoft.com: "Use WM_INPUT here and in wMsgFilterMax to specify only the WM_INPUT messages."
//    if GetMessage(lpMsg, getInput.fRawInputWindowHnd, WM_INPUT, WM_INPUT) then
//      DispatchMessage(lpMsg);
//  until False;


////////////////////////////////////////////////////////////////////////////////
// approach #3: PeekMessage()
// >> comment-out other approaches; comment-out last DefWindowProc() line in RawInputWndProc(), leave case block

//  repeat
//    if PeekMessage(lpMsg, getInput.fRawInputWindowHnd, WM_INPUT, WM_INPUT, PM_NOREMOVE) then
//      DispatchMessage(lpMsg);
//
//    if PeekMessage(lpMsg, 0, 0, 0, PM_REMOVE) then
//      DispatchMessage(lpMsg);
//  until False;

  getInput.Free();
end.

Answer 1

I've overhauled this 'answer' based on the exchange in the comments below and involved testing. It does not necessarily answer my question but represents my current level of understanding and outlines the approach I ended up taking (and which seems to be working so far)

• RawInput seems to be sent through WM_INPUT window messages in any case; whether when using GetRawInputData() or GetRawInputBuffer()
• This means some kind of window is needed to which the messages can be sent to. This can be a hidden window. Using CreateWindowEx(0, PChar('Message'), nil, 0, 0, 0, 0, 0, HWND_MESSAGE, 0, 0, nil); works very well for me so far
• This also means that there needs to be a message loop of some kind so messages can be worked off (and don't pile up).
• The difference to GetRawInputData() seems to be that Windows will 'queue up' WM_INPUT messages and GetRawInputBuffer() gets and removes (from the queue) multiple messages at once. And I think the single advantage there is that input can be 'received in' quicker (higher throughput) this way than having to 'deal with every WM_INPUT message individually'.
• What's tricky is that it seems like for GetRawInputBuffer() to work, it's paramount that messages except WM_INPUT are handled by regular means - and then GetRawInputBuffer() gets called regularly, which deals with the queued-up WM_INPUT messages. Any approach I took which in some way 'looked' at WM_INPUT messages ultimately caused me to get inconsistent / incomplete results from GetRawInputBuffer()

Below is my message loop, which is largely inspired by this SO answer and runs in a separate thread

repeat
  TThread.Sleep(10);

  while True do
  begin
    if (Not PeekMessage(lpMsg, 0, 0, WM_INPUT - 1, PM_NOYIELD or PM_REMOVE)) then System.Break;
    DefWindowProc(lpMsg.hwnd, lpMsg.message, lpMsg.wParam, lpMsg.lParam);
  end;

  while True do
  begin
    if (Not PeekMessage(lpMsg, 0, WM_INPUT + 1, High(Cardinal), PM_NOYIELD or PM_REMOVE)) then System.Break;
    DefWindowProc(lpMsg.hwnd, lpMsg.message, lpMsg.wParam, lpMsg.lParam);
  end;

  ReadRawInputBuffer();     // shown below; essentially reads out all queued-up input
until SomeCondition;

Reading the buffer (largely inspired by the sample code on learn.microsoft.com):

procedure ReadInputBuffer();
var
  // ...

begin
  // this returns the minimum required buffer size in ```pcbSize```
  numberOfStructs := GetRawInputBuffer(nil, pcbSize, rawInputHeaderSize);
  if (numberOfStructs = 0) then
  begin
    // read out all queued-up data
    repeat
      // ... allocate pBuffer as needed
      numberOfStructs := GetRawInputBuffer(pBuffer, pcbSize, rawInputHeaderSize);
      if ((numberOfStructs > 0) and (numberOfStructs < 900000)) then
        // do something with pBuffer / its data
        // I use a TThreadedQueue<T>; the items/data is worked off outside this thread
      else
        System.Break;
    until False;
  end
end;

(In tests of over 10 minutes, reading in > 700'000 key events doesn't seem to have lost me a single one (if my numbers don't lie). Using TStopWatch and starting/stopping at the start of the message loop (after TThread.Sleep(10)) and stopping at the end after having exhausted the input queue, in one test reading about 12k events in 15 seconds (that's close to 800 events per second), the slowest run measured... 0ms.)