Using Java
If your client is a Java client (which you don't say in your question), you could try the following code from this answer:
Using the Apache Commons IO library...
boolean isFileUnlocked = false;
try {
org.apache.commons.io.FileUtils.touch(yourFile);
isFileUnlocked = true;
} catch (IOException e) {
isFileUnlocked = false;
}
if(isFileUnlocked){
// Do stuff you need to do with a file that is NOT locked.
} else {
// Do stuff you need to do with a file that IS locked
}
BTW: This is the very first answer when you search for "java file is open" on Google.
Using JavaScript
However, I assume that you don't have a Java client, only a Java server and you want to upload a file via a webpage (so you have a HTML + JS client). I assume this, because you don't tell us.
JavaScript doesn't allow you to write to the disk since this would be a security issue (I think Chrome supports some kind of the JavaScript File API which allows you to write to the disk, but none of the other browsers). So you cannot test this via JavaScript.
In general
The general way to test if a file is open, would be to test if there is some lock on the file. But this would not guarantee that the file is not opened without a lock, but it should be sufficient.