Winapi - JNA implementation of native WinProc for Eclipse SWT GUI Window is having issues

Question

As an extension to my recent question (Winapi - SetWindowLongPtr in ShutdownBlockReasonCreate / Destroy implementation of JNI native code) I would like to know if there's any chance to implement the same capability using JNA (latest release 5.5.0 - https://github.com/java-native-access/jna).

Since I couldn't find anything relates to SetWindowSubclass() in the doc (http://java-native-access.github.io/jna/5.5.0/javadoc/) I had to use SetWindowLongPtr().

After doing some research online, here're some of my code snippets responsible for the intended functionality:

    private static LONG_PTR baseWndProc;
    private static HWND hWnd;

    public static void initiateWindowsShutdownHook() {
        Display.getDefault().syncExec(()->{
            hWnd = new WinDef.HWND(Pointer.createConstant(Display.getDefault().getActiveShell().handle));
        });

        baseWndProc = IWindowsAPIUtil.WSBINSTANCE.SetWindowLongPtr(
                hWnd, IWindowsAPIUtil.GWL_WNDPROC, new IWindowsAPIUtil.WNDPROC() {

            @Override
            public LRESULT callback(HWND hWnd, int uMsg, WPARAM wParam, LPARAM lParam) {
                switch(uMsg) {
                case IWindowsAPIUtil.WM_QUERYENDSESSION:
                    Logger.logWarning("Shutdown initiated");
                    IWindowsAPIUtil.WSBINSTANCE.PostMessage(hWnd, User32.WM_CLOSE, wParam, lParam);
                    return new LRESULT(0);
                }
                return IWindowsAPIUtil.WSBINSTANCE.CallWindowProc(baseWndProc, hWnd, uMsg, wParam, lParam);
            }
        });
    }


    public interface IWindowsAPIUtil extends User32 {

        public static final IWindowsAPIUtil WSBINSTANCE = 
                (IWindowsAPIUtil) Native.loadLibrary("user32", IWindowsAPIUtil.class, W32APIOptions.UNICODE_OPTIONS);

        interface WNDPROC extends StdCallCallback{
            LRESULT callback(HWND hWnd, int uMsg, WPARAM wParam, LPARAM lParam);
        }

        public static final int GWL_WNDPROC = -4;
        public static final int WM_QUERYENDSESSION = 17;

        LONG_PTR SetWindowLongPtr(HWND hWnd, int nIndex, WNDPROC wndProc);
        LONG_PTR GetWindowLongPtr(HWND hWnd, int nIndex);
        LRESULT CallWindowProc(LONG_PTR proc, HWND hWnd, int uMsg, WPARAM wParam, LPARAM lParam);
        void PostMessage(HWND hWnd, int uMsg, WPARAM wParam, LPARAM lParam);
    }

And my new c++ native code now looks like this:

Note: Since in this exercise I only refactored the callback part (in original c++ native code) into JNA so no need for SetWindowSubclass() here

#include <windows.h>

#include <jni.h>

#include <iostream>
#include "com_app_project_winapi_WindowsAPI.h"


#include <commctrl.h>

using namespace std;

namespace {
    // Default reason text. The actual reason text should be defined by application logic not the native code
    LPCWSTR SHUTDOWN_REASON = L"Application is still saving ...";
}


JNIEXPORT void JNICALL Java_com_app_project_winapi_WindowsAPI_shutdownBlockReasonCreate(JNIEnv *env, jclass cls, jstring title, jstring reasonText) {
    cout << "In shutdownBlockReasonCreate method" << endl;

    const jchar *str = env->GetStringChars(title, NULL);
    HWND hWnd = FindWindowW(NULL, (LPCWSTR)str);
    env->ReleaseStringChars(title, str);
    if (hWnd == NULL) {
        return;
    }

    ShutdownBlockReasonCreate(hWnd, SHUTDOWN_REASON);

    return;
}

JNIEXPORT void JNICALL Java_com_app_project_winapi_WindowsAPI_shutdownBlockReasonDestroy(JNIEnv *env, jclass cls, jstring title) {
    cout << "In shutdownBlockReasonDestroy method" << endl;

    const jchar *str = env->GetStringChars(title, NULL);
    HWND hWnd = FindWindowW(NULL, (LPCWSTR)str);
    env->ReleaseStringChars(title, str);
    if (hWnd == NULL) {
        return;
    }

    ShutdownBlockReasonDestroy(hWnd);

    return;
}

From my code you can probably see I use Eclipse SWT for my GUI application. After saving my code and run the application I get the following issue:

1. While blockReasonCreate is activated during the first application save (intent of this capability is to block shutdown when saving), it no longer activates in subsequent saves. And reason text says "This app is preventing shutdown", instead of the reason text that was passed in.
2. As an extended behaviour from above, my GUI application freezes and the window could not be closed until I force close it from Task Manager

I tried the following:

1. Replace "baseWndProc" LONG_PTR with GetWindowLongPtr() in CallWindowProc(). Unfortunately didn't work
2. I have a suspicion I'm having the same problem with SetWindowLongPtr() as last time. But as described earlier JNA does not seemed to have the matching SetWindowSubclass() method provided so I'm out of ideas.

Btw, the native code solution from last time still works perfectly. But for maintainability purposes getting all functionality implemented in Java would be ideal.

Really appreciated anyone taking time on my issue!

Cheers

Answer 1

You should always prepare a minimal sample, that reproduces the problem and can be run by potential helpers. Reasoning over code, that is ripped from context is difficult.

In your case, you are creating the callback as an anonymous class (the new IWindowsAPIUtil.WNDPROC() /* ... */ in your sample). This callback immediately becomes eligible for garbage collection after the call to SetWindowLongPtr returns. The JVM will not run a GC immediatly, but it will.

It is not defined what the OS/JVM will do when you try to call into a callback, that was GCed. The JNA documentation is clear here:

If native code attempts to call a callback which has been GC'd, you will likely crash the VM. If there is no method to deregister the callback (e.g. atexit in the C library), you must ensure that you always keep a live reference to the callback object.

So you should keep a strong reference to the callback on the java side (for example use a static field, that will have the same lifetime like the window).