Scala: Building a complex hierarchy of traits and classes

Question

I have posted several questions on SO recently dealing with Scala traits, representation types, member types, manifests, and implicit evidence. Behind these questions is my project to build modeling software for biological protein networks. Despite the immensely helpful answers, which have gotten me closer than I ever could get on my own, I have still not arrived at an solution for my project. A couple of answers have suggested that my design is flawed, which is why the solutions to the Foo-framed questions don't work in practice. Here I am posting a more complicated (but still greatly simplified) version of my problem. My hope is that the problem and solution will be broadly useful for people trying to build complex hierarchies of traits and classes in Scala.

The highest-level class in my project is the biological reaction rule. A rule describes how one or two reactants are transformed by a reaction. Each reactant is a graph that has nodes called monomers and edges that connect between named sites on the monomers. Each site also has a state that it can be in. Edit: The concept of the edges have been removed from the example code because they complicate the example without contributing much to the question. A rule might say something like this: there is one reactant made of monomer A bound to monomer B through sites a1 and b1, respectively; the bond is broken by the rule leaving sites a1 and b1 unbound; simultaneously on monomer A, the state of site a1 is changed from U to P. I would write this as:

A(a1~U-1).B(b1-1) -> A(a1~P) + B(b1)

(Parsing strings like this in Scala was so easy, it made my head spin.) The -1 indicates that bond #1 is between those sites--the number is just a arbitrary label.

Here is what I have so far along with the reasoning for why I added each component. It compiles, but only with gratuitous use of asInstanceOf. How do I get rid of the asInstanceOfs so that the types match?

I represent rules with a basic class:

case class Rule(
  reactants: Seq[ReactantGraph], // The starting monomers and edges
  producedMonomers: Seq[ProducedMonomer] // Only new monomers go here
) {
  // Example method that shows different monomers being combined and down-cast
  def combineIntoOneGraph: Graph = {
    val all_monomers = reactants.flatMap(_.monomers) ++ producedMonomers
    GraphClass(all_monomers)
  }
}

The class for graphs GraphClass has type parameters because so that I can put constraints on what kinds of monomers and edges are allowed in a particular graph; for example, there cannot be any ProducedMonomers in the Reactant of a Rule. I would also like to be able to collect all the Monomers of a particular type, say ReactantMonomers. I use type aliases to manage the constraints.

case class GraphClass[
  +MonomerType <: Monomer
](
  monomers: Seq[MonomerType]
) {
  // Methods that demonstrate the need for a manifest on MonomerClass
  def justTheProductMonomers: Seq[ProductMonomer] = {
    monomers.collect{
      case x if isProductMonomer(x) => x.asInstanceOf[ProductMonomer]
    }
  }
  def isProductMonomer(monomer: Monomer): Boolean = (
    monomer.manifest <:< manifest[ProductStateSite]
  )
}

// The most generic Graph
type Graph = GraphClass[Monomer]
// Anything allowed in a reactant
type ReactantGraph = GraphClass[ReactantMonomer]
// Anything allowed in a product, which I sometimes extract from a Rule
type ProductGraph = GraphClass[ProductMonomer]

The class for monomers MonomerClass has type parameters, as well, so that I can put constraints on the sites; for example, a ConsumedMonomer cannot have a StaticStateSite. Furthermore, I need to collect all the monomers of a particular type to, say, collect all the monomers in a rule that are in the product, so I add a Manifest to each type parameter.

case class MonomerClass[
  +StateSiteType <: StateSite : Manifest
](
  stateSites: Seq[StateSiteType]
) {
  type MyType = MonomerClass[StateSiteType]
  def manifest = implicitly[Manifest[_ <: StateSiteType]]

  // Method that demonstrates the need for implicit evidence
  // This is where it gets bad
  def replaceSiteWithIntersection[A >: StateSiteType <: ReactantStateSite](
    thisSite: A, // This is a member of this.stateSites
    monomer: ReactantMonomer
  )(
    // Only the sites on ReactantMonomers have the Observed property
    implicit evidence: MyType <:< ReactantMonomer
  ): MyType = {
    val new_this = evidence(this) // implicit evidence usually needs some help
    monomer.stateSites.find(_.name == thisSite.name) match {
      case Some(otherSite) => 
        val newSites = stateSites map {
          case `thisSite` => (
            thisSite.asInstanceOf[StateSiteType with ReactantStateSite]
            .createIntersection(otherSite).asInstanceOf[StateSiteType]
          )
          case other => other
        }
        copy(stateSites = newSites)
      case None => this
    }
  }
}

type Monomer = MonomerClass[StateSite]
type ReactantMonomer = MonomerClass[ReactantStateSite]
type ProductMonomer = MonomerClass[ProductStateSite]
type ConsumedMonomer = MonomerClass[ConsumedStateSite]
type ProducedMonomer = MonomerClass[ProducedStateSite]
type StaticMonomer = MonomerClass[StaticStateSite]

My current implementation for StateSite does not have type parameters; it is a standard hierarchy of traits, terminating in classes that have a name and some Strings that represent the appropriate state. (Be nice about using strings to hold object states; they are actually name classes in my real code.) One important purpose of these traits is provide functionality that all the subclasses need. Well, isn't that the purpose of all traits. My traits are special in that many of the methods make small changes to a property of the object that is common to all subclasses of the trait and then return a copy. It would be preferable if the return type matched the underlying type of the object. The lame way to do this is to make all the trait methods abstract, and copy the desired methods into all the subclasses. I am unsure of the proper Scala way to do this. Some sources suggest a member type MyType that stores the underlying type (shown here). Other sources suggest a representation type parameter.

trait StateSite {
  type MyType <: StateSite 
  def name: String
}
trait ReactantStateSite extends StateSite {
  type MyType <: ReactantStateSite
  def observed: Seq[String]
  def stateCopy(observed: Seq[String]): MyType
  def createIntersection(otherSite: ReactantStateSite): MyType = {
    val newStates = observed.intersect(otherSite.observed)
    stateCopy(newStates)
  }
}
trait ProductStateSite extends StateSite
trait ConservedStateSite extends ReactantStateSite with ProductStateSite 
case class ConsumedStateSite(name: String, consumed: Seq[String]) 
  extends ReactantStateSite {
  type MyType = ConsumedStateSite
  def observed = consumed
  def stateCopy(observed: Seq[String]) = copy(consumed = observed)
}
case class ProducedStateSite(name: String, Produced: String)
  extends ProductStateSite 
case class ChangedStateSite(
  name: String, 
  consumed: Seq[String], 
  Produced: String
)
  extends ConservedStateSite {
  type MyType = ChangedStateSite
  def observed = consumed
  def stateCopy(observed: Seq[String]) = copy(consumed = observed)
}
case class StaticStateSite(name: String, static: Seq[String])
  extends ConservedStateSite {
  type MyType = StaticStateSite
  def observed = static
  def stateCopy(observed: Seq[String]) = copy(static = observed)
}

My biggest problems are with methods framed like MonomerClass.replaceSiteWithIntersection. A lot of methods do some complicated search for particular members of the class, then pass those members to other functions where complicated changes are made to them and return a copy, which then replaces the original in a copy of the higher-level object. How should I parameterize methods (or the classes) so that the calls are type safe? Right now I can get the code to compile only with lots of asInstanceOfs everywhere. Scala is particularly unhappy with passing instances of a type or member parameter around because of two main reasons that I can see: (1) the covariant type parameter ends up as input to any method that takes them as input, and (2) it is difficult to convince Scala that a method that returns a copy indeed returns an object with exactly the same type as was put in.

I have undoubtedly left some things that will not be clear to everyone. If there are any details I need to add, or excess details I need to delete, I will try to be quick to clear things up.

Edit

@0__ replaced the replaceSiteWithIntersection with a method that compiled without asInstanceOf. Unfortunately, I can't find a way to call the method without a type error. His code is essentially the first method in this new class for MonomerClass; I added the second method that calls it.

case class MonomerClass[+StateSiteType <: StateSite/* : Manifest*/](
  stateSites: Seq[StateSiteType]) {
  type MyType = MonomerClass[StateSiteType]
  //def manifest = implicitly[Manifest[_ <: StateSiteType]]

  def replaceSiteWithIntersection[A <: ReactantStateSite { type MyType = A }]
    (thisSite: A, otherMonomer: ReactantMonomer)
    (implicit ev: this.type <:< MonomerClass[A])
  : MonomerClass[A] = {
    val new_this = ev(this)

    otherMonomer.stateSites.find(_.name == thisSite.name) match {
      case Some(otherSite) =>
        val newSites = new_this.stateSites map {
          case `thisSite` => thisSite.createIntersection(otherSite)
          case other      => other
        }
        copy(stateSites = newSites)
      case None => new_this // This throws an exception in the real program
    }
  }

  // Example method that calls the previous method
  def replaceSomeSiteOnThisOtherMonomer(otherMonomer: ReactantMonomer)
      (implicit ev: MyType <:< ReactantMonomer): MyType = {
    // Find a state that is a current member of this.stateSites
    // Obviously, a more sophisticated means of selection is actually used
    val thisSite = ev(this).stateSites(0)

    // I can't get this to compile even with asInstanceOf
    replaceSiteWithIntersection(thisSite, otherMonomer)
  }
}

Answer 1

I have reduced your problem to traits, and I am starting to understand why you are getting into troubles with casts and abstract types.

What you are actually missing is ad-hoc polymorphism, which you obtain through the following: - Writing a method with generic signature relying on an implicit of the same generic to delegate the work to - Making the implicit available only for specific value of that generic parameter, which will turn into a "implicit not found" compile time error when you try to do something illegal.

Let's now look to the problem in order. The first is that the signature of your method is wrong for two reasons:

• When replacing a site you want to create a new monomer of the new generic type, much as you do when you add to a collection an object which is a superclass of the existing generic type: you get a new collection whose type parameter is the superclass. You should yield this new Monomer as a result.

• You are not sure that the operation will yield a result (in case you can't really replace a state). In such a case the right type it's Option[T]

  def replaceSiteWithIntersection[A >: StateSiteType <: ReactantStateSite]
  (thisSite: A, monomer: ReactantMonomer): Option[MonomerClass[A]] 
  

If we now look digger in the type errors, we can see that the real type error comes from this method:

 thisSite.createIntersection

The reason is simple: it's signature is not coherent with the rest of your types, because it accepts a ReactantSite but you want to call it passing as parameter one of your stateSites (which is of type Seq[StateSiteType] ) but you have no guarantee that

StateSiteType<:<ReactantSite

Now let's see how evidences can help you:

trait Intersector[T] {
  def apply(observed: Seq[String]): T
}


trait StateSite {

  def name: String
}

trait ReactantStateSite extends StateSite {

  def observed: Seq[String]

  def createIntersection[A](otherSite: ReactantStateSite)(implicit intersector: Intersector[A]): A = {
    val newStates = observed.intersect(otherSite.observed)
    intersector(newStates)
  }
}


import Monomers._
trait MonomerClass[+StateSiteType <: StateSite] {

    val stateSites: Seq[StateSiteType]



    def replaceSiteWithIntersection[A >: StateSiteType <: ReactantStateSite](thisSite: A, otherMonomer: ReactantMonomer)(implicit intersector:Intersector[A], ev: StateSiteType <:< ReactantStateSite): Option[MonomerClass[A]] = {

      def replaceOrKeep(condition: (StateSiteType) => Boolean)(f: (StateSiteType) => A)(implicit ev: StateSiteType<:<A): Seq[A] = {
        stateSites.map {
                         site => if (condition(site)) f(site) else site
                       }
      }


      val reactantSiteToIntersect:Option[ReactantStateSite] = otherMonomer.stateSites.find(_.name == thisSite.name)
      reactantSiteToIntersect.map {
               siteToReplace =>
               val newSites = replaceOrKeep {_ == thisSite } { item => thisSite.createIntersection( ev(item) ) }
               MonomerClass(newSites)
             }


    }


  }

object MonomerClass {
  def apply[A <: StateSite](sites:Seq[A]):MonomerClass[A] =  new MonomerClass[A] {
    val stateSites = sites
  }
}
object Monomers{

  type Monomer = MonomerClass[StateSite]
  type ReactantMonomer = MonomerClass[ReactantStateSite]
  type ProductMonomer = MonomerClass[ProductStateSite]
  type ProducedMonomer = MonomerClass[ProducedStateSite]

}

1. Please note that this pattern can be used with no special imports if you use in a clever way implicit resolving rules (for example you put your insector in the companion object of Intersector trait, so that it will be automatically resolved).

2. While this pattern works perfectly, there is a limitation connected to the fact that your solution works only for a specific StateSiteType. Scala collections solve a similar problem adding another implicit, which is call CanBuildFrom. In our case we will call it CanReact

You will have to make your MonomerClass invariant, which might be a problem though (why do you need covariance, however?)

trait CanReact[A, B] {
  implicit val intersector: Intersector[B]

  def react(a: A, b: B): B

  def reactFunction(b:B) : A=>B = react(_:A,b)
}

object CanReact {

  implicit def CanReactWithReactantSite[A<:ReactantStateSite](implicit inters: Intersector[A]): CanReact[ReactantStateSite,A] = {
    new CanReact[ReactantStateSite,A] {
      val intersector = inters

      def react(a: ReactantStateSite, b: A) = a.createIntersection(b)
    }
  }
}

trait MonomerClass[StateSiteType <: StateSite] {

    val stateSites: Seq[StateSiteType]



    def replaceSiteWithIntersection[A >: StateSiteType <: ReactantStateSite](thisSite: A, otherMonomer: ReactantMonomer)(implicit canReact:CanReact[StateSiteType,A]): Option[MonomerClass[A]] = {

      def replaceOrKeep(condition: (StateSiteType) => Boolean)(f: (StateSiteType) => A)(implicit ev: StateSiteType<:<A): Seq[A] = {
        stateSites.map {
                         site => if (condition(site)) f(site) else site
                       }
      }


      val reactantSiteToIntersect:Option[ReactantStateSite] = otherMonomer.stateSites.find(_.name == thisSite.name)
      reactantSiteToIntersect.map {
               siteToReplace =>
               val newSites = replaceOrKeep {_ == thisSite } { canReact.reactFunction(thisSite)}
               MonomerClass(newSites)
             }


    }


  }

With such an implementation, whenever you want to make the possibility to replace a site with another site of a different type, all you need is to make available new implicit instances of CanReact with different types.

I will conclude with a (I hope) clear explanation of why you should not need covariance.

Let's say you have a Consumer[T] and a Producer[T].

You need covariance when you want to provide to the Consumer[T1] a Producer[T2] where T2<:<T1 . But if you need to use the value produced by T2 inside T1, you can

class ConsumerOfStuff[T <: CanBeContained] {

  def doWith(stuff: Stuff[T]) = stuff.t.writeSomething

}

trait CanBeContained {
  def writeSomething: Unit
}

class A extends CanBeContained {
  def writeSomething = println("hello")
}


class B extends A {
  override def writeSomething = println("goodbye")
}

class Stuff[T <: CanBeContained](val t: T)

object VarianceTest {

  val stuff1 = new Stuff(new A)
  val stuff2 = new Stuff(new B)
  val consumerOfStuff = new ConsumerOfStuff[A]
  consumerOfStuff.doWith(stuff2)

}

This stuff clearly not compiles:

error: type mismatch; found : Stuff[B] required: Stuff[A] Note: B <: A, but class Stuff is invariant in type T. You may wish to define T as +T instead. (SLS 4.5) consumerOfStuff.doWith(stuff2).

But again, this come from a misinterpretation of usage of variance, as How are co- and contra-variance used in designing business applications? Kris Nuttycombe answer explain. If we refactor like the following

class ConsumerOfStuff[T <: CanBeContained] {

  def doWith[A<:T](stuff: Stuff[A]) = stuff.t.writeSomething

}

You could see everything compiling fine.

Answer 2

Not an answer, but what I can observe from looking over the question:

• I see MonomerClass but not Monomer

My guts say you should avoid manifests when possible, as you have seen they can make things complicated. I don't think you will need them. For example the justTheProductMonomers method in GraphClass – since you have complete control over your class hierarchy, why not add test methods for anything involving runtime checks to Monomer directly? E.g.

trait Monomer {
   def productOption: Option[ProductMonomer]
}

then you'll have

def justTheProductMonomers : Seq[ProductMonomer] = monomers.flatMap( _.productOption )

and so forth.

The problem here is that it seems you can have a generic monomer satisfying the product predicate, while you somehow want sub-type ProductMonomer.

The general advise I would give is first to define your matrix of tests that you need to process the rules, and then put those tests as methods into the particular traits, unless you have a flat hierarchy for which you can do pattern matching, which is easier since the disambiguation will appear concentrated at your use site, and not spread across all implementing types.

Also don't try to overdue it with compile-time type constraints. Often it's perfectly fine to have some constraints checked at runtime. That way at least you can construct a fully working system, and then you can try to spot the points where you can convert a runtime check into a compile time check, and decide whether the effort is worth it or not. It is appealing to solve things on the type level in Scala, because of its sophistication, but it also requires the most skills to do it right.

Answer 3

There are multiple problems. First, the whole method is weird: On the one hand you passing in a monomer argument, and if the argument thisState is found, the method has nothing to do with the receiver—then why is this a method in MonomerClass at all and not a "free floating" function—, on the other hand you fall back to returning this if thisSite is not found. Since you originally had also implicit evidence: MyType <:< ReactantMonomer, my guess is the whole monomer argument is obsolete, and you actually wanted to operate on new_this.

A bit of cleanup, forgetting the manifests for the moment, you could have

case class MonomerClass[+StateSiteType <: StateSite, +EdgeSiteType <: EdgeSite](
  stateSites: Seq[StateSiteType], edgeSites: Seq[EdgeSiteType]) {

  def replaceSiteWithIntersection[A <: ReactantStateSite { type MyType = A }]
  (thisSite: A)(implicit ev: this.type <:< MonomerClass[A, ReactantEdgeSite])
  : MonomerClass[A, ReactantEdgeSite] = {
    val monomer = ev(this)
    monomer.stateSites.find(_.name == thisSite.name) match {
      case Some(otherSite) => 
        val newSites = monomer.stateSites map {
          case `thisSite` => thisSite.createIntersection(otherSite)
          case other      => other
        }
        monomer.copy(stateSites = newSites)
      case None => monomer
    }
  }
}

This was an interesting problem, it took me some iterations to get rid of the (wrong!) casting. Now it is actually quite readable: This method is restricted to the evidence that StateSiteType is actually a subtype A of ReactantStateSite. Therefore, the type parameter A <: ReactantStateSite { type MyType = A }—the last bit is interesting, and this was a new find for myself: You can specify the type member here to make sure that your return type from createIntersection is actually A.

---

There is still something odd with your method, because if I'm not mistaken, you will end up calling x.createIntersection(x) (intersecting thisSite with itself, which is a no-op).

Answer 4

One thing that is flawed about replaceSiteWithIntersection is that according to the method signature the type of thisSite (A) is a super-type of StateSiteType and a sub-type of ReactantStateSite.

But then you eventually cast it to StateSiteType with ReactantStateSite. That doesn't make sense to me.

Where do you get the assurance from that A suddenly is a StateSiteType?