You could use a Area
for example...

public class TestPane extends JPanel {
public TestPane() {
}
@Override
public Dimension getPreferredSize() {
return new Dimension(200, 200);
}
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
Graphics2D g2d = (Graphics2D) g.create();
Area area = new Area(new Rectangle(10, 10, getWidth() - 20, getHeight() - 20));
area.subtract(new Area(new Rectangle(20, getHeight() / 2, getWidth() / 2, getHeight() - 10)));
g2d.draw(area);
g2d.dispose();
}
}
You define a custom shape...

public class TestPane extends JPanel {
public TestPane() {
}
@Override
public Dimension getPreferredSize() {
return new Dimension(200, 200);
}
@Override
protected void paintComponent(Graphics g) {
super.paintComponent(g);
Graphics2D g2d = (Graphics2D) g.create();
Path2D path = new Path2D.Float();
path.moveTo(10, 10);
path.lineTo(getWidth() - 20, 10);
path.lineTo(getWidth() - 20, getHeight() - 20);
path.lineTo(getWidth() / 2, getHeight() - 20);
path.lineTo(getWidth() / 2, getHeight() / 2);
path.lineTo(20, getHeight() / 2);
path.lineTo(20, getHeight() - 20);
path.lineTo(10, getHeight() - 20);
path.closePath();
g2d.draw(path);
g2d.dispose();
}
}
Actually writing a custom border would be very, very difficult, because of the irregular style of shape, where would the components actually be contained?
It might be possible to create two or more borders, which could then be laid out so that the appeared as one
See Working with Geometry for more details
Updated with Border
example...
Getting a Border
to actually work is far more difficult, as the expectation is that the internal area of the border will be rectangular.
Based on the complex shape you've provided, one solution would be to actually create two borders, a left and right borer, which take care of generating a "safe" area for components to be laid out within, for example:
public class LeftBorder implements Border {
private int offset;
public LeftBorder(int offset) {
this.offset = offset;
}
@Override
public void paintBorder(Component c, Graphics g, int x, int y, int width, int height) {
Path2D path = new Path2D.Float();
int xOffset = x + offset;
int yOffset = y + offset;
width -= offset;
height -= offset * 2;
float gap = width * 0.1f;
path.moveTo(xOffset, yOffset);
path.lineTo(xOffset + width, yOffset);
path.moveTo(xOffset, yOffset);
path.lineTo(xOffset, yOffset + height);
path.lineTo(xOffset + gap, yOffset + height);
path.lineTo(xOffset + gap, yOffset + (height - (height / 2)));
path.lineTo(xOffset + width, yOffset + (height - (height / 2)));
((Graphics2D)g).draw(path);
}
@Override
public Insets getBorderInsets(Component c) {
int height = c.getHeight();
height -= (height / 2);
System.out.println(height);
return new Insets(offset + 4, offset + 4, height + 4, 0);
}
@Override
public boolean isBorderOpaque() {
return false;
}
}
public class RightBorder implements Border {
private int offset;
public RightBorder(int offset) {
this.offset = offset;
}
@Override
public void paintBorder(Component c, Graphics g, int x, int y, int width, int height) {
Path2D path = new Path2D.Float();
int xOffset = x;
int yOffset = y + offset;
width -= offset;
height -= offset * 2;
path.moveTo(xOffset, yOffset);
path.lineTo(xOffset + width, yOffset);
path.lineTo(xOffset + width, yOffset + height);
path.lineTo(xOffset, yOffset + height);
path.lineTo(xOffset, yOffset + (height - (height / 2)));
((Graphics2D)g).draw(path);
}
@Override
public Insets getBorderInsets(Component c) {
return new Insets(offset + 4, 0, offset + 4, offset + 4);
}
@Override
public boolean isBorderOpaque() {
return false;
}
}
This would then require you to provide at least two panels of equal height, for example:

import java.awt.Component;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.GridBagConstraints;
import java.awt.GridBagLayout;
import java.awt.Insets;
import java.awt.geom.Path2D;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.UIManager;
import javax.swing.UnsupportedLookAndFeelException;
import javax.swing.border.Border;
public class Main {
public static void main(String args[]) {
new Main();
}
public Main() {
EventQueue.invokeLater(new Runnable() {
@Override
public void run() {
try {
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
ex.printStackTrace();
}
JFrame frame = new JFrame("Testing");
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setLayout(new GridBagLayout());
frame.add(new LeftPane());
frame.add(new RightPane());
frame.pack();
frame.setLocationRelativeTo(null);
frame.setVisible(true);
}
});
}
public class RightPane extends JPanel {
public RightPane() {
setBorder(new RightBorder(10));
setLayout(new GridBagLayout());
add(new JLabel("Righty"));
}
@Override
public Dimension getPreferredSize() {
return new Dimension(200, 200);
}
}
public class LeftPane extends JPanel {
public LeftPane() {
setBorder(new LeftBorder(10));
setLayout(new GridBagLayout());
add(new JLabel("Lefty"));
}
@Override
public Dimension getPreferredSize() {
return new Dimension(200, 200);
}
}
}
This will also be relient on the layout manager been able to layout the two components next to each other