I don't seem to have any issues...
public class TestLayeredDialog {
public static void main(String[] args) {
new TestLayeredDialog();
}
public TestLayeredDialog() {
EventQueue.invokeLater(new Runnable() {
@Override
public void run() {
try {
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
}
JDialog dialog = new JDialog();
dialog.setModal(true);
dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
dialog.setLayout(new BorderLayout());
dialog.add(new MyContent());
dialog.pack();
dialog.setLocationRelativeTo(null);
dialog.setVisible(true);
System.exit(0);
}
});
}
public class MyContent extends JLayeredPane {
public MyContent() {
JLabel label = new JLabel("Hello new world");
label.setSize(label.getPreferredSize());
label.setLocation(0, 0);
add(label);
Dimension size = getPreferredSize();
JButton button = new JButton("Click me");
button.setSize(button.getPreferredSize());
button.setLocation(size.width - button.getWidth(), size.height - button.getHeight());
button.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
SwingUtilities.getWindowAncestor(MyContent.this).dispose();
}
});
add(button);
}
@Override
public Dimension getPreferredSize() {
return new Dimension(200, 200);
}
}
}
Remember, JLayeredPane
DOES NOT have a layout manager. You become responsible for managing the size and position of the child components, that's the point.
Updated with new example

public class TestLayeredDialog {
public static void main(String[] args) {
new TestLayeredDialog();
}
public TestLayeredDialog() {
EventQueue.invokeLater(new Runnable() {
@Override
public void run() {
try {
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
}
JDialog dialog = new JDialog();
dialog.setModal(true);
dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
dialog.setLayout(new BorderLayout());
JLabel label = new JLabel("Hello new world");
label.setSize(label.getPreferredSize());
label.setLocation(0, 0);
dialog.getLayeredPane().add(label, new Integer(1));
dialog.setSize(100, 100);
dialog.setLocationRelativeTo(null);
dialog.setVisible(true);
System.exit(0);
}
});
}
}
The layered pane of the JRootPane
is responsible for (amongst other things) laying out the content pane and menu bar. It is also used (in some cases) to display things like popups.

Have a read through How to Use Root Panes
You can choose to put components in the root pane's layered pane. If
you do, then you should be aware that certain depths are defined to be
used for specific functions, and you should use the depths as
intended. Otherwise, your components might not play well with the
others. Here's a diagram that shows the functional layers and their
relationship:
Using this, means you are competing with components already on the screen.
Unless you have VERY good reason to be messing with this component, I would suggest you avoid it as 1- It's possible to be changed in the future (the layer position of the components) and 2- It may interfere with other components used by the Swing API