If you use a Mutex, logically that Mutex would need to be accessible from any JVM which was running a copy of "the program". In C programming, this might be accomplished via shared memory, but Java doesn't have such a thing by default.
With that understanding, there are plenty of ways to implement what you want. You could open a server socket on a designated port (the operating system assures that only one process is the recipient of the server socket, and subsequent opens fail).
You could use a "lock file" but it is a bit complicated, as the file you would need to use would really be a directory (and it becomes heavily dependent on whether directory creation is atomic for your file system, even though most directory creations are). If a sysadmin decides to run you via NFS, then things get even harder (if not impossible).
You can also do a number of nifty tricks with JVMs and debugging / JMI, provided you can somehow assure youself that all relevant JVMs are launched with the same configurations (in time, an impossible task).
Other people have used the exec facility to run the equivalent of a process listing, but it is a bit tricky due to the possibility of race condition (two processes simultaneously check, and fail to see each other).
In the end, the server socket route is probably the most stable, as it is guaranteed to only bind to one process by the TCP/IP stack (and is mediated by the operating system). That said, you will have to flush the socket of incoming messages, and it opens up the possibility of other security issues.