Most idiomatic way to mix synchronous, asynchronous, and parallel computation in a scala for comprehension of futures

Question

Suppose I have 4 future computations to do. The first two can be done in parallel, but the third must be done after the first two (even though the values of the first two are not used in the third -- think of each computation as a command that performs some db operation). Finally, there is a 4th computation that must occur after all of the first 3. Additionally, there is a side effect that can be started after the first 3 complete (think of this as kicking off a periodic runnable). In code, this could look like the following:

for {
  _ <- async1     // not done in parallel with async2 :( is there
  _ <- async2     // any way of achieving this cleanly inside of for?
  _ <- async3
  _ =  sideEffect // do I need "=" here??
  _ <- async4
} yield ()

The comments show my doubts about the quality of the code:

1. What's the cleanest way to do two operations in parallel in a for comprehension?
2. Is there is a way to achieve this result without so many "_" characters (nor assigning a named reference, at least in the case of sideEffect)
3. what's the cleanest and most idiomatic way to do this?

Answer 1

You can use zip to combine two futures, including the result of zip itself. You'll end up with tuples holding tuples, but if you use infix notation for Tuple2 it is easy to take them apart. Below I define a synonym ~ for succinctness (this is what the parser combinator library does, except its ~ is a different class that behaves similiarly to Tuple2).

As an alternative for _ = for the side effect, you can either move it into the yield, or combine it with the following statement using braces and a semicolon. I would still consider _ = to be more idiomatic, at least so far as having a side effecting statement in the for is idiomatic at all.

val ~ = Tuple2

for {
  a ~ b ~ c <- async1 zip
               async2 zip
               async3
  d <- { sideEffect; async4 }
} yield (a, b, c, d)

Answer 2

for-comprehensions represent monadic operations, and monadic operations are sequenced. There's superclass of monad, applicative, where computations don't depend on the results of prior computations, thus may be run in parallel.

Scalaz has a |@| operator for combining applicatives, so you can use (future1 |@| future2)(proc(_, _)) to dispatch two futures in parallel and then run "proc" on the result of both of them, as opposed to sequential computation of for {a <- future1; b <- future2(a)} yield b (or just future1 flatMap future2).

There's already a method on stdlib Futures called .zip that combines Futures in parallel, and indeed the scalaz impl uses this: https://github.com/scalaz/scalaz/blob/scalaz-seven/core/src/main/scala/scalaz/std/Future.scala#L36 And .zip and for-comprehensions may be intermixed to have parallel and sequential parts, as appropriate. So just using the stdlib syntax, your above example could be written as:

for {
  _ <- async1 zip async2
  _ <- async3
  _ =  sideEffect
  _ <- async4
} yield ()

Alternatively, written w/out a for-comprehension:

async1 zip async2 flatMap (_=> async3) flatMap {_=> sideEffect; async4}

Answer 3

Just as an FYI, it's really simple to get two futures to run in parallel and still process them via a for-comprehension. The suggested solutions of using zip can certainly work, but I find that when I want to handle a couple of futures and do something when they are all done, and I have two or more that are independent of each other, I do something like this:

val f1 = async1
val f2 = async2
//First two futures now running in parallel

for {
  r1 <- f1     
  r2 <- f2     
  _ <- async3
  _ =  sideEffect 
  _ <- async4
} yield {
  ...
}

Now the way the for comprehension is structured certainly waits on f1 before checking on the completion status of f2, but the logic behind these two futures is running at the same time. This is a little simpler then some of the suggestions but still might give you what you need.

Answer 4

Your code already looks structured minus computing futures in parallel.

1. Use helper functions, ideally writing a code generator to print out helpers for all tuple cases
2. As far as I know, you need to name the result or assign it _
3. Example code

Example code with helpers.

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

object Example {
  def run: Future[Unit] = {
    for {
      (a, b, c) <- par(
        Future.successful(1),
        Future.successful(2),
        Future.successful(3)
      )
      constant = 100
      (d, e) <- par(
        Future.successful(a + 10),
        Future.successful(b + c)
      )
    } yield {
      println(constant)
      println(d)
      println(e)
    }
  }

  def par[A,B](a: Future[A], b: Future[B]): Future[(A, B)] = {
    for {
      a <- a
      b <- b
    } yield (a, b)
  }

  def par[A,B,C](a: Future[A], b: Future[B], c: Future[C]): Future[(A, B, C)] = {
    for {
      a <- a
      b <- b
      c <- c
    } yield (a, b, c)
  }
}
Example.run

Edit:

generated code for 1 to 20 futures: https://gist.github.com/nanop/c448db7ac1dfd6545967#file-parhelpers-scala

parPrinter script: https://gist.github.com/nanop/c448db7ac1dfd6545967#file-parprinter-scala