If you're just starting the project and haven't actually got the JMF part up and running yet, then you might want to take a look at some alternatives before committing to it.
If you want to go ahead with Swing, to get the general overlay behaviour you want, you'll need to make use of Frame.setUndecorated()
to turn off window borders and buttons, and Window.setAlwaysOnTop()
to make sure the window stays on top of other windows. For the transparency, see this tutorial. However, I'm not sure whether transparency and video will work nicely together, so good luck!
You might also want to write a custom focus handler for the window so that it cannot be focused, although it is probably impossible for the overlay to be properly 'phantom' whereby clicks just pass through the overlay to the underlying desktop. That kind of behaviour might only be possible by using low-level graphics techniques i.e. by not creating a window at all, but by drawing directly onto the screen. That might require a native library.