Scala generic type usage in Apache Spark Dataset creation

Question

The following code creates a empty Dataset in Spark..

    scala> val strings = spark.emptyDataset[String]
    strings: org.apache.spark.sql.Dataset[String] = [value: string]

The signature of emptyDataset is..

     @Experimental  
     @InterfaceStability.Evolving
     def emptyDataset[T: Encoder]: Dataset[T] = {
         val encoder = implicitly[Encoder[T]]
         new Dataset(self, LocalRelation(encoder.schema.toAttributes), encoder)   
     }

Why the above signature is not forcing T to be subtype of Encoder?

It accepts T as of type String and creates a encoder for String and pass that to Dataset constructor.It finally creates Dataset[String].

score 5 · Answer 1 · answered Mar 14 '17 at 09:29

This syntax is actually a syntactic sugar for requiring an implicit Encoder[T]. The following functions are equivalent:

def foo[A : Encoder](a: A)

def foo[A](a: A)(implicit encoder: Encoder[A])

The syntax for subtyping is actually defined by type bounds, something like A <: B for example. I hope that helps you.

Scala generic type usage in Apache Spark Dataset creation

1 Answers1