The best you can do in pure Java is to use the Robot
class to capture a specified rectangle. The following Q&A show how ... for the case where the rectangle is the entire screen.
(This should work for all non-headless Java SE platforms.)
That's the easy bit. The hard bit is figuring out what rectangle to screenshot. The Java libraries provide no APIs for finding information about the screen locations of "other" applications. To find that kind of information, your Java application would need to interact with the native OS via a native library or an external utility command. This will inevitably be platform specific ... meaning that it will break for Mac and Linux, and possibly for other versions of Windows as well.
I wouldn't try to do this in Java at all. Instead, I would look for an OS-specific screenshot application that can be run as an external application; e.g. using Process.exec. Alternatively, I'd figure out how to write such an app, in (say) C#.
I think getting the application window from the title of the window may be a start, but does Java even have access to the GUIs of other processes?
Nope and nope.