The Problem
Before offering a solution I think it's important, or at least interesting, to understand why having a TextFormatter seems to change the behavior of the Dialog. If this doesn't matter to you, feel free to jump to the end of the answer.
Cancel Buttons
According to the documentation of Button, a cancel button is:
the button that receives a keyboard VK_ESC press, if no other node in the scene consumes it.
The end of that sentence is the important part. The way cancel buttons, as well as default buttons, are implemented is by registering an accelerator with the Scene that the Button belongs to. These accelerators are only invoked if the appropriate KeyEvent bubbles up to the Scene. If the event is consumed before it reaches the Scene, the accelerator is not invoked.
Note: To understand more about event processing in JavaFX, especially terms such as "bubbles" and "consumed", I suggest reading this tutorial.
Dialogs
A Dialog has certain rules regarding how and when it can be closed. These rules are documented here, in the Dialog Closing Rules section. Suffice to say, basically everything depends on which ButtonTypes have been added to the DialogPane. In your example you use one of the predefined types: ButtonType.CANCEL. If you look at the documentation of that field, you'll see:
A pre-defined ButtonType that displays "Cancel" and has a ButtonBar.ButtonData of ButtonBar.ButtonData.CANCEL_CLOSE.
And if you look at the documentation of ButtonData.CANCEL_CLOSE, you'll see:
A tag for the "cancel" or "close" button.
Is cancel button: True
What this means, at least for the default implementation, is that the Button created for said ButtonType.CANCEL will be a cancel button. In other words, the Button will have its cancelButton property set to true. This is what allows one to close a Dialog by pressing the Esc key.
Note: It's the DialogPane#createButton(ButtonType) method that's responsible for creating the appropriate button (and can be overridden for customization). While the return type of that method is Node it is typical, as documented, to return an instance of Button.
The TextFormatter
Every control in (core) JavaFX has three components: the control class, the skin class, and the behavior class. The latter class is responsible for handling user input, such as mouse and key events. In this case, we care about TextInputControlBehavior and TextFieldBehavior; the former is the superclass of the latter.
Note: Unlike the skin classes, which became public API in JavaFX 9, the behavior classes are still private API as of JavaFX 12.0.2. Much of what's described below are implementation details.
The TextInputControlBehavior class registers an EventHandler that reacts to the Esc key being pressed, invoking the cancelEdit(KeyEvent) method of the same class. All the base implementation of this method does is forward the KeyEvent to the TextInputControl's parent, if it has one—resulting in two event dispatching cycles for some unknown (to me) reason. However, the TextFieldBehavior class overrides this method:
@Override
protected void cancelEdit(KeyEvent event) {
TextField textField = getNode();
if (textField.getTextFormatter() != null) {
textField.cancelEdit();
event.consume();
} else {
super.cancelEdit(event);
}
}
As you can see, the presence of a TextFormatter causes the KeyEvent to be unconditionally consumed. This stops the event from reaching the Scene, the cancel button is not fired, and thus the Dialog does not close when the Esc key is pressed while the TextField has the focus. When there is no TextFormatter the super implementation is invoked which, as stated before, simply forwards the event to the parent.
The reason for this behavior is hinted at by the call to TextInputControl#cancelEdit(). That method has a "sister method" in the form of TextInputControl#commitValue(). If you look at the documentation of those two methods, you'll see:
If the field is currently being edited, this call will set text to the last commited value.
And:
Commit the current text and convert it to a value.
Respectively. That doesn't explain much, unfortunately, but if you look at the implementation their purpose becomes clear. A TextFormatter has a value property which is not updated in real time while typing into the TextField. Instead, the value is only updated when it's committed (e.g. by pressing Enter). The reverse is also true; the current text can be reverted to the current value by cancelling the edit (e.g. by pressing Esc).
Note: The conversion between String and an object of arbitrary type is handled by the StringConverter associated with the TextFormatter.
When there's a TextFormatter, the act of cancelling the edit is deemed an event-consuming scenario. This makes sense, I suppose. However, even when there's nothing to cancel the event is still consumed—this doesn't make as much sense to me.
A Solution
One way to fix this is to dig into the internals, using reflection, as is shown in kleopatra's answer. Another option is to add an event filter to the TextField or some ancestor of the TextField that closes the Dialog when the Esc key is pressed.
textField.addEventFilter(KeyEvent.KEY_PRESSED, event -> {
if (event.getCode() == KeyCode.ESCAPE) {
event.consume();
dialog.close();
}
});
If you'd like to include the cancel-edit behavior (cancel without closing) then you should only close the Dialog if there's no edit to cancel. Take a look at kleopatra's answer to see how one might determine whether or not a cancel is needed. If there is something to cancel simply don't consume the event and don't close the Dialog. If there isn't anything to cancel then just do the same as the code above (i.e. consume and close).
Is using an event filter the "recommended way"? It's certainly a valid way. JavaFX is event-driven like most, if not all, mainstream UI toolkits. For JavaFX specifically that means reacting to Events or observing Observable[Value]s for invalidations/changes. A framework built "on top of" JavaFX may add its own mechanisms. Since the problem is an event being consumed when we don't want it to be, it is valid to add your own handlers to implement the desired behavior.
