If you want to connect from a separate java process, consider using MatlabControl. It allows you to easily launch new Matlab instance from java process and then communicate with it though JMI, i.e. java-to-matlab interface that is shipped with each Matlab release.
Is it also possible to connect to existing Matlab instance, but you will need to modify MatlabControl code to do that, and you will need to manually execute some command from within the Matlab instance you wish to connect to.
There is a nice series of articles on MatlabControl / JMI on Undocumented Matlab.
As of 2016b, there is also an official MATLAB Engine API for Java.