TraversableOnce, Future, and Option in a Scala for comprehension

Question

I have a list of string ids representing DB records. I'd like to load them from the DB asynchronously, then upload each record to a remote server asynchronously, then when all are done uploading, make a record of the ids of the records that were uploaded.

Since I'm on Scala 2.9.2, I'm using Twitter's core-util Future implementation, but it should work exactly like the 2.10 futures in terms of Monadic transformations.

The general concept is this:

def fetch(id: String): Future[Option[Record]]
def upload(record: Record): Future[String]
def notifyUploaded(ids: Seq[String]): Unit

val ids: Seq[String] = ....

I'm trying to do this via a for comprehension but the fact that fetch returns a Future of Option makes it obscure and the code doesn't compile:

for {
  id <- ids
  maybeRecord <- fetch(id)
  record <- maybeRecord
  uploadedId <- upload(record)
} yield uploadedId

Compiling this results in the following error:

scala: type mismatch;
found   : com.twitter.util.Future[String]
required: Option[?]
    uploadedId <- upload(record)
                  ^

What am I missing? why does the compiler expect uploadedId to be an Option? is there any pretty way I could work around this?

possible duplicate of [Type Mismatch on Scala For Comprehension](http://stackoverflow.com/questions/4719592/type-mismatch-on-scala-for-comprehension) — om-nom-nom, Mar 01 '13 at 09:21
A Monad is a Monoid in the category of **Endo**-functors. Just sayin' — George, Mar 01 '13 at 09:34
@folone: I fear that not everyone will get the [joke](http://stackoverflow.com/questions/3870088/a-monad-is-just-a-monoid-in-the-category-of-endofunctors-whats-the-problem). Just sayin' — Régis Jean-Gilles, Mar 01 '13 at 10:06
@folone I thought that all Functors in Scala are Endofunctors (in the category of Scala types), so why the emphasis on **Endo** in this particular case? — Ben James, Mar 01 '13 at 10:46
@BenJames yeah, but the author's using three of them in one comprehension, right? — George, Mar 01 '13 at 11:19
@folone But the issue is that the natural transformation is from the product of two endofunctors to itself, not that they themselves are endofunctors. — Impredicative, Mar 01 '13 at 11:34
@Impredicative I'm definitely confusing the terminology here. Am I wrong stating that using, say, two different functors in for comprehension makes you break the "in the category of endofunctors" part? — George, Mar 01 '13 at 11:47
@folone So, each monad is an object in the category of endofunctors on the category of Scala types. In that category, the morphisms are natural transformations between functors, and the thing being broken is that the natural transformation for 'flatMap/bind' is from `M x M -> M`, whereas composing it would be something like `M x T -> TM` (e.g. the composition of the functors.) The problem is that it's not guaranteed (and often not true) that the composition is itself a monad. — Impredicative, Mar 01 '13 at 11:57
@Impredicative oh, so this is actually about the monad composition. I see now, thanks for your explanation! — George, Mar 01 '13 at 12:08

Impredicative · Answer 1 · 2013-03-01T10:23:09.707

Consider the signature of the flatMap (or bind) function:

trait Monad[M[_]] {
  def flatMap[A](a : M[A], f : A => M[B]) : M[B]
  ....

In your case, you're trying to use flatMap on an Option, giving it an f that generates a Future. But as in the signature above, f should be generating something in the same monad that it's been called on.

Scala isn't necessarily terribly helpful in this regard, since it's pretty good at converting things around (to Seqs, for example) in such a way that you get the impression that you can chain arbitrary flatMap calls together, regardless of the container.

What you possibly want is a 'Monad transformer', which gives you some ability to compose monads. Debasish Ghosh has a post on using Scalaz monad transformers here.

score 0 · Answer 2 · answered Sep 22 '16 at 17:55

You cannot mix all different types in one for comprehension, I figured out that you might mix Seq and Option and result would be either Seq or Option depending on what is first. It is not possible to mix Future and Seq or Option. If you want to use for-comprehension you would have to cascade them few. In such cases it might be nicer with map/flatMap. I implemented your question in both ways and added types to few intermediate results so that you see the mess that is being created while working with all that different types.

object TestClass {

  import scala.concurrent.Future
  import scala.concurrent.ExecutionContext.Implicits.global
  import scala.concurrent._
  import scala.concurrent.duration._

  case class Record(id: String)


  def fetch(id: String): Future[Option[Record]] = Future {
    Thread.sleep(1000);
    Some(Record(id))
  }

  def upload(record: Record): Future[String] = Future {
    Thread.sleep(3000);
    record.id + "_uploaded"
  }

  def notifyUploaded(ids: Seq[String]): Unit = println("notified" + ids)

  val ids: Seq[String] = Seq("a", "b", "c")

  def main(args: Array[String]): Unit = {
    forComprehensionImpl()
    mapAndFlatMapImpl()
  }

  def forComprehensionImpl() = {
    val result: Seq[Future[Option[Future[String]]]] = for {
      id <- ids
    } yield {
      for {
        maybeRecord <- fetch(id)
      } yield {
        for {
          record <- maybeRecord
        } yield {
          for {
            uploadedId <- upload(record)
          } yield uploadedId
        }
      }
    }
    val result2: Future[Seq[Option[Future[String]]]] = Future.sequence(result)
    val result3: Future[Unit] = result2.flatMap { x: Seq[Option[Future[String]]] =>
      Future.sequence(x.flatten).map(notifyUploaded)
    }
    Await.result(result3, Duration.Inf)
  }


  def mapAndFlatMapImpl() = {
    val res: Seq[Future[Iterable[String]]] = ids.map { id =>
      fetch(id).flatMap { maybeRecord =>
        val res1: Option[Future[Seq[String]]] = maybeRecord.map { record =>
          upload(record) map (Seq(_))
        }
        res1 match {
          case Some(a) => a
          case None => Future(Seq())
        }
      }
    }
    val res3: Future[Unit] = Future.sequence(res) map (a => notifyUploaded(a.flatten))
    Await.result(res3, Duration.Inf)
  }
}

TraversableOnce, Future, and Option in a Scala for comprehension

2 Answers2