How to discover that a Scala remote actor is died?

Question

In Scala, an actor can be notified when another (remote) actor terminates by setting the trapExit flag and invoking the link() method with the second actor as parameter. In this case when the remote actor ends its job by calling exit() the first one is notified by receiving an Exit message.

But what happens when the remote actor terminates in a less graceful way (e.g. the VM where it is running crashes)? In other words, how the local actor can discover that the remote one is no longer available? Of course I would prefer (if possible) that the local actor could be notified by a message similar to the Exit one, but it seems not feasible. Am I missing something? Should I continuously polling the state of the remote actor (and in this case I don't know which is the best way to do that) or is there a smarter solution?

Answer 1

The book Actors in Scala mentions (not tested personally):

Trapping termination notifications.

In some cases, it is useful to receive termination notifications as messages in the mailbox of a monitoring actor.
For example, a monitoring actor may want to rethrow an exception that is not handled by some linked actor.
Or, a monitoring actor may want to react to normal termination, which is not possible by default.

Actors can be configured to receive all termination notifications as normal messages in their mailbox using the Boolean trapExit flag. In the following example actor b links itself to actor a:

val a = actor { ... }
val b = actor {
self.trapExit = true
link(a)
...
}

Note that before actor b invokes link it sets its trapExit member to true;
this means that whenever a linked actor terminates (normally or abnormally) it receives a message of type Exit.
Therefore, actor b is going to be notified whenever actor a terminates (assuming that actor a did not terminate before b’s invocation of link).

So "what happens when the remote actor terminates in a less graceful way"?
It should receive an Exit message even in the case of an abnormal termination.

val b = actor {
  self.trapExit = true
  link(a)
  a ! 'start
  react {
    case Exit(from, reason) if from == a =>
    println("Actor 'a' terminated because of " + reason)
  }
}

Answer 2

But what happens when the remote actor terminates in a less graceful way (e.g. the VM where it is running crashes)

Actor proxy stays alive accepting messages (and loosing them), and waiting for you to restart the JVM with remote actor. Watching for JVM crashes (and other failures happening on the infrastructure level) is far beyond Scala responsibilities. Good choice for that could be monitoring through JMX.

In other words, how the local actor can discover that the remote one is no longer available?

You may define a timeout interval (say 5000 millis). If remote actor doesn't reply during this interval, it's a sign for you that something unexpected is happening to remote actor, and you may either ask it about its state or just treat it as dead.

Should I continuously polling the state of the remote actor (and in this case I don't know which is the best way to do that) or is there a smarter solution?

You may put a kind of a polling load balancer/dispatcher in front of a group of actors, that will use only those actors that are alive and ready to process messages (which makes sense in case of remote actors that may suddenly appear/disappear behind the proxy) -> Can Scala actors process multiple messages simultaneously?