Have you considered a CardLayout? http://docs.oracle.com/javase/tutorial/uiswing/layout/card.html
Personally, I find the use of a separate dialogue to be a bit dated for configuration settings. I prefer tabbed layouts, which are card layouts decorated with a tab bar across the top.
You could easily wrap your application in a near-top-level card layout and add a menu action to switch to the configuration card, with the "acknowledgement" or "cancel" buttons switching back to the main application card.
In the end, it is really about what your users prefer, but remember a lot of them might prefer what they know, even if it is not a better solution. You have to find a balance, and if your implementation rocks, then eventually they will want your approach to the problem to be used in other applications.
A perfect example of this is tabbed browsing, as opposed to multiple windows. Personally, I can't imagine going back to multiple-window browsing now that I have become accustomed to browsing tabs, but at one point in time, multiple windows was the only game in town.
In the end, if you find out you made the wrong choice, keep you code clean enough to easily implement with either solution. As long as your configuration screen is just a plain JPanel (or wrapped in just a JPanel), it shouldn't be very hard to do.