Oook, so I managed to solve it though I'm not exactly pleased with the solution. At least there is no mystery and it works :).
The reason why I didn't receive the hotkeys Ctrl+Alt+F11 and Ctrl+Alt+F12
They were registered as global hotkeys. I managed to find this out using the ActiveHotkeys program of the fellow stackoverflow member moodforaday (thanks a lot for it!). Apparently there is no documented way to find out which program registered a particular hotkey (and it didn't do anything on my system). See moodforaday's thread about the issue.
The solution
One of the answers in the aforementioned thread led me to another question. Efotinis' answer was absolutely perfect for me. I did not have experience with setting up low-level keyboard hooks, but it was not nearly as difficult as it sounds. For future's sake, here is how I did it in my Qt application:
In my mainwindow.h:
class MainWindow : public QMainWindow
{
Q_OBJECT
public:
explicit MainWindow(QWidget *parent = 0);
~MainWindow();
// ... code
private:
void tryLogin();
friend LRESULT CALLBACK LowLevelKeyboardProc(int code, WPARAM wparam, LPARAM lparam);
};
LRESULT CALLBACK LowLevelKeyboardProc(int code, WPARAM wparam, LPARAM lparam);
In my mainwindow.cpp:
// setting up the hook in the constructor
SetWindowsHookEx(WH_KEYBOARD_LL,
LowLevelKeyboardProc,
NULL,
0);
The hook code (mostly from efotinis' answer):
LRESULT CALLBACK LowLevelKeyboardProc(int code, WPARAM wparam, LPARAM lparam)
{
KBDLLHOOKSTRUCT* kllhs = reinterpret_cast<KBDLLHOOKSTRUCT*>(lparam);
if (code == HC_ACTION)
{
if (wparam == WM_KEYDOWN && kllhs->vkCode == VK_F12 &&
(GetAsyncKeyState(VK_MENU) < 0 && GetAsyncKeyState(VK_CONTROL) < 0))
{
MainWindow* w = dynamic_cast<MainWindow*>(qApp->activeWindow());
if (NULL != w)
{
w->tryLogin(); // this should not be blocking!
return 1;
}
}
}
return CallNextHookEx(0, code, wparam, lparam);
}
As you can see, we get the pointer to the application window from the global QApplication object. We use dynamic_cast so in the active window happens to not be a MainWindow instance we would get a NULL pointer.
If you are wondering why the GetAsyncKeyState calls are checked for < 0, it's because this function returns with MSB set if the key is down. And when the MSB is set, the SHORT number is negative (on x86/x64 and compatible platforms). If windows ever gets ported to a platform where signed integers are represented differently, this code might break. The absolutely proper way would be to create a 16-bit mask and use that to check the MSB, but I'm lazy to do that. :)
One thing to note is that when you call a function from your hook, the Qt event loop has just begun processing. That means until you don't return from your function, it will block the UI (freezing it for a few seconds). If you wanted to show a dialog like I did, instead of exec(), call raise, activateWindow and show, while setting the window modality of the dialog to modal (in its constructor maybe).
You can unregister the hook with UnHookWindowsHookEx if you want to (which happens when the module containing the hook is unloaded). To do this, save the return value of the SetWindowsHookEx call.
