This doesn't appear to be possible. I was going to suggest a basic loop that would iterate open windows and set all those other than the dialog to a different cursor:
JDialog dialog = new JDialog(ownerFrame, "dialog", true); // modal
dialog.setLocationRelativeTo(ownerFrame);
Cursor cursorInvalid;
try {
cursorInvalid = Cursor.getSystemCustomCursor("Invalid.32x32");
} catch (HeadlessException | AWTException e) {
throw new RuntimeException(e);
}
Map<Window,Cursor> previousCursors = new HashMap<>();
for (Window w : Window.getWindows()) {
if (w != dialog) {
previousCursors.put(w, w.getCursor());
w.setCursor(cursorInvalid);
}
}
dialog.setVisible(true);
for (Window w : Window.getWindows()) {
w.setCursor(previousCursors.get(w));
}
This does change the cursor property of the windows successfully, but while the modal dialog is open, then the cursor property of those windows is apparently ignored and they always display a default arrow. I also tried setting the cursor property of individual components of the window, but again, as soon as the modal dialog is opened, the cursor property is ignored.
I tested this on Windows. The behavior is quite possibly platform-dependent, so it might work on other operating systems.
However, even if it works, I'd like to suggest that it's not a good idea do it. It risks confusion to defy user's expectations of how basic windows and dialogs work. The operating system already has its own rules for that. On Windows for example, attempting to interact with a window blocked by a modal dialog causes a beep sound and the modal dialog's title bar is flashed, and that is well-understood behavior.