Splitting a Monix Observable

Question

I would like to write a split function for monix.reactive.Observable. It should split a source Observable[A] into a new pair (Observable[A], Observable[A]), based on the value of a predicate, evaluated against each element in the source. I would like the split to work independently of whether the source Observable is hot or cold. In the case where the source is cold, the new pair of Observables should also be cold and where the source is hot the new pair of Observables will be hot. I would like to know if such an implementation is possible and, if so, how (I have pasted a failing testcase below).

The signature, as a method on an implicit class, would look like, or similar to

    /**
      * Split an observable by a predicate, placing values for which the predicate returns true
      * to the right (and values for which the predicate returns false to the left).
      * This is consistent with the convention adopted by Either.cond.
      */
    def split(p: T => Boolean)(implicit scheduler: Scheduler, taskLike: TaskLike[Future]): (Observable[T], Observable[T]) = {
      splitEither[T, T](elem => Either.cond(p(elem), elem, elem))
    }

Currently, I have a naive implementation that consumes the source elements and pushes them to PublishSubject. The new pair of Observables is thus hot. My tests for a cold Observable are failing.

import monix.eval.TaskLike
import monix.execution.{Ack, Scheduler}
import monix.reactive.{Observable, Observer}
import monix.reactive.subjects.PublishSubject

import scala.concurrent.Future

object ObservableOps {

  implicit class ObservableExtensions[T](o: Observable[T]) {

    /**
      * Split an observable by a predicate, placing values for which the predicate returns true
      * to the right (and values for which the predicate returns false to the left).
      * This is consistent with the convention adopted by Either.cond.
      */
    def split(p: T => Boolean)(implicit scheduler: Scheduler, taskLike: TaskLike[Future]): (Observable[T], Observable[T]) = {
      splitEither[T, T](elem => Either.cond(p(elem), elem, elem))
    }

    /**
      * Split an observable into a pair of Observables, one left, one right, according
      * to a determinant function.
      */
    def splitEither[U, V](f: T => Either[U, V])(implicit scheduler: Scheduler, taskLike: TaskLike[Future]): (Observable[U], Observable[V]) = {
      val l = PublishSubject[U]()
      val r = PublishSubject[V]()

      o.subscribe(new Observer[T] {
        override def onNext(elem: T): Future[Ack] = {
          f(elem) match {
            case Left(u) => l.onNext(u)
            case Right(v) => r.onNext(v)
          }
        }

        override def onError(ex: Throwable): Unit = {
          l.onError(ex)
          r.onError(ex)
        }

        override def onComplete(): Unit = {
          l.onComplete()
          r.onComplete()
        }
      })

      (l, r)
    }
  }
}

//////////


import ObservableOps._

import monix.execution.Scheduler.Implicits.global
import monix.reactive.Observable
import monix.reactive.subjects.PublishSubject

import org.scalatest.FlatSpec
import org.scalatest.Matchers._
import org.scalatest.concurrent.ScalaFutures._

class ObservableOpsSpec extends FlatSpec {

  val isEven: Int => Boolean = _ % 2 == 0

  "Observable Ops" should "split a cold observable" in {
    val o = Observable(1, 2, 3, 4, 5)

    val (l, r) = o.split(isEven)

    l.toListL.runToFuture.futureValue shouldBe List(1, 3, 5)
    r.toListL.runToFuture.futureValue shouldBe List(2, 4)
  }

  "Observable Ops" should "split a hot observable" in {
    val o = PublishSubject[Int]()

    val (l, r) = o.split(isEven)
    val lbuf = l.toListL.runToFuture
    val rbuf = r.toListL.runToFuture

    Observable.fromIterable(1 to 5).mapEvalF(i => o.onNext(i)).subscribe()
    o.onComplete()

    lbuf.futureValue shouldBe List(1, 3, 5)
    rbuf.futureValue shouldBe List(2, 4)
  }
}

I expect both testcases above to pass but "Observable Ops" should "split a cold observable" is failing.

Edit: working code

An implementation that passes both test cases is as follows:

import monix.execution.Scheduler
import monix.reactive.Observable

object ObservableOps {

  implicit class ObservableExtension[T](o: Observable[T]) {

    /**
      * Split an observable by a predicate, placing values for which the predicate returns true
      * to the right (and values for which the predicate returns false to the left).
      * This is consistent with the convention adopted by Either.cond.
      */
    def split(
        p: T => Boolean
    )(implicit scheduler: Scheduler): (Observable[T], Observable[T]) = {
      splitEither[T, T](elem => Either.cond(p(elem), elem, elem))
    }

    /**
      * Split an observable into a pair of Observables, one left, one right, according
      * to a determinant function.
      */
    def splitEither[U, V](
        f: T => Either[U, V]
    )(implicit scheduler: Scheduler): (Observable[U], Observable[V]) = {

      val oo = o.map(f)

      val l = oo.collect {
        case Left(u) => u
      }

      val r = oo.collect {
        case Right(v) => v
      }

      (l, r)
    }
  }
}

score 2 · Answer 1 · answered Sep 12 '19 at 15:51

class ObservableOpsSpec extends FlatSpec {

  val isEven: Int => Boolean = _ % 2 == 0

  "Observable Ops" should "split a cold observable" in {
    val o = Observable(1, 2, 3, 4, 5)
   val o2 =  o.publish
    val (l, r) = o2.split(isEven)

  val x=   l.toListL.runToFuture
  val y =   r.toListL.runToFuture
    o2.connect()
    x.futureValue shouldBe List(1, 3, 5)
    y.futureValue shouldBe List(2, 4)
  }

  "Observable Ops" should "split a hot observable" in {
    val o = PublishSubject[Int]()

    val (l, r) = o.split(isEven)
    val lbuf = l.toListL.runToFuture
    val rbuf = r.toListL.runToFuture

    Observable.fromIterable(1 to 5).mapEvalF(i => o.onNext(i)).subscribe()
    o.onComplete()

    lbuf.futureValue shouldBe List(1, 3, 5)
    rbuf.futureValue shouldBe List(2, 4)
  }
}

Converting the cold Observable into a hot one with `Observable.publish` means it still does not work for the cold case. — Jeremy Townson, Sep 12 '19 at 19:36

score 2 · Answer 2 · answered Sep 12 '19 at 20:03

2

Cold observable, by definition, is lazily evaluated for each subscriber. You can't split it without either evaluating everything twice or converting it into hot one.

If you don't mind evaluating everything twice, just use .filter two times. If you don't mind converting to hot, do it with .publish (or .publish.refCount so you don't need to connect manually). If you want to preserve cold/hot property and process two pieces in parallel, there's a publishSelector method that lets you treat any observable like a hot one in a limited scope:

coldOrHot.publishSelector { totallyHot =>
  val s1 = totallyHot.filter(...).flatMap(...) // any processing
  val s2 = totallyHot.filter(...).mapEval(...) // any processing 2
  Observable(s1, s2).merge
}

It's limitation, apart from scope, is that result of inner lambda has to be another Observable (which will be returned from publishSelector), so you can't have the helper with the signature you want. But the result will still be cold if the original was cold.

answered Sep 12 '19 at 20:03

Oleg Pyzhcov

7,323
1
18
30

I have added an implementation that uses collect twice. – Jeremy Townson Sep 13 '19 at 11:51
1

@JeremyTownson Your tests are insufficient. Here's a [scastie](https://scastie.scala-lang.org/QDLKU23xTMCKPyw16OroKA) demonstrating that it does, in fact, evaluate everything twice. – Oleg Pyzhcov Sep 13 '19 at 13:16
Yes, I was resigned to that, even if a single evaluation would be better. It is a pity that such seemingly simple operations are so difficult. Creating complex data pipelines should be as simple as connecting up plumbing or electrical extension leads, but is not. It feels to me like monix is missing a layer. – Jeremy Townson Sep 17 '19 at 08:13
@JeremyTownson it's how lazy/cold streams work - you can't *just* separate them. This is a conceptual limitation that you might run into, irrespective of library. – Oleg Pyzhcov Sep 18 '19 at 15:46

Splitting a Monix Observable

Edit: working code

2 Answers2