RMI's speed is governed by two things:
- Default Java serialization
- Distributed garbage collection
Default Java serialization can be surprisingly bloated. You can make your object serialization more lightweight by implementing Externalizale
and doing your own bare-bones serialization. This already start to look like doing a custom protocol.
Distributed Garbage Collection can become a factor if your system grows large and contains many JVMs. DGC involves JVMs exchanging messages alerting each other about objects which are subject to garbage collection. It can potentially create a lot of network traffic.
That said, RMI "out of the box" can be faster than other "out of the box" alternatives. For example, SOAP can be much less efficient on the wire and involves a much deeper and heavier network stack than RMI.
You can build a faster custom RPC than RMI, but if you rely on Java serialization, it probably won't be much faster because of point #1 above.
Finally, why do you want a faster protocol? Are you having problems with the speed of RMI? Are you looking to pick the fastest "out of the box" solution upfront? Keep in mind The rules of Optimize Club.