Option generator in for comprehension forces other generators to be Option?

Question

The first for expression is the base example. The 2nd for-expr introduces a slight variation which I thought would yield the same output. But failed with a compilation error. What is the reason and how to fix?

for {
    n <- List(1,2)
    c <- "ABC"
} yield s"$c$n"
//res0: List[String] = List(A1, B1, C1, A2, B2, C2)

for {
    opt <- List(None, Some(1), None, None, Some(2), None)
    n <- opt
    c <- "ABC"
} yield s"$c$n"
//Error:(14, 5) type mismatch;
//found   : scala.collection.immutable.IndexedSeq[String]
//required: Option[?]
//  c <- "ABC"
//     ^

In this case yes because the second extractor is like a `flatMap` so you need to return an option. For clarity, you are able to extract on "ABC" in the first case because of an implicit conversion, so that thing becomes a list and it compiles, in the second case "ABC" is not implicitly converted to option. — Ende Neu, Oct 04 '16 at 12:51

score 2 · Accepted Answer · edited May 23 '17 at 12:15

To reply to the title question: yes, the first generator "sets the mood" for the whole for-expression. Note that for-expressions like the ones above are desugared into calls to flatMaps and a final map (plus withFilter for guards).

In your case, the first for-expression is desugared into the following expression:

List(1, 2).flatMap(n => "ABC".map(c => s"$c$n"))

This can work because Predef (which is implicitly imported in every Scala program) provides an implicit conversion of a String to a Seq[Char].

val implicitlyConverted: Seq[Char] = "ABC"

Thus, it type checks an run as planned.

Now let's give a look at how the second for-expressions is desugared:

List(None, Some(1), None, None, Some(2), None).flatMap(opt => opt.flatMap(n => "ABC".map(c => s"$c$n")))

Again, we have the same type error as above, if we break the expression into several lines we may see the issue a little bit better:

List(None, Some(1), None, None, Some(2), None).flatMap {
  opt =>
    opt.flatMap {
      n =>
        "ABC".map {
          c =>
            s"$c$n"
        }
    }
}

This gives us the following error:

<console>:12: error: type mismatch;
 found   : scala.collection.immutable.IndexedSeq[String]
 required: Option[?]
                      "ABC".map {
                                ^

The second flatMap expects an Option[_] as a result, while the map within ("ABC".map(...)) returns an IndexedSeq[String].

So, this was the reason. How do we fix this? The easiest solution probably involves using a guard and forcibly extracting the value within the Option, like this:

for {
  n <- List(None, Some(1), None, None, Some(2), None)
  c <- "ABC" if n.isDefined
} yield s"$c${n.get}"

A more general way to solve this particular issue involves monad transformers, because the root of the issue is that monads do not compose; the issue is very wide and complex and perhaps this reply may give you a more general answer.

The academic quality of the answer is above expectation. Thank you very much. — Polymerase, Oct 06 '16 at 20:05

score 2 · Answer 2 · answered Oct 04 '16 at 13:50

Another way to "fix" it is to change the order:

for {
    opt <- List(None, Some(1), None, None, Some(2), None)
    c <- "ABC"
    n <- opt
} yield s"$c$n"
//> List[String] = List(A1, B1, C1, A2, B2, C2)

It works because Scala can convert Option to a List, but not in the opposite direction.

In fact it's much more complicated than simple conversion and involves CanBuildFrom. Normally monads do not compose at all, but in Scala some do by using CanBuildFrom.

Option generator in for comprehension forces other generators to be Option?

2 Answers2