28

I've recently started learning Scala and was disappointed (but not surprised) that their generics are also implemented via type erasure.

My question is, is it possible for Scala to have reified generics, or would the JVM need to be changed in some way? If the JVM does need to be changed, what exactly would need to be changed?

cdmckay
  • 31,832
  • 25
  • 83
  • 114

5 Answers5

21

No - it is not possible for Scala to run as Java-equivalent bytecode if that bytecode does not support reified generics.

When you ask "what is it that needs to be changed?", the answer is: the bytecode specification. Currently the bytecode does not allow for the parametrized type of a variable to be defined. It has been decided that as a modification to the bytecode to support reified generics would break backwards compatibility, that generics would have to be implemented via type erasure.

In order to get around this, Scala has used the power of its implicit mechanism to define a Manifest which can be imported in any scope to discover type information at runtime. Manifests are experimental and largely undocumented but they are coming as part of the library in 2.8. Here is another good resource on Scala reified generics / Manifests

oxbow_lakes
  • 133,303
  • 56
  • 317
  • 449
  • Links being added as you speak :-) – oxbow_lakes Aug 31 '09 at 15:22
  • I had no idea the Scala guys were doing this. This is very cool. – cdmckay Aug 31 '09 at 15:32
  • Small correction: Manifests have been an experimental part of the standard library since 2.7.2. I don't know whether they will be any less experimental in 2.8. – Jorge Ortiz Aug 31 '09 at 17:11
  • Thanks *Jorge*. I was going from the current 2.8 scaladoc which includes manifests. – oxbow_lakes Aug 31 '09 at 18:14
  • 4
    IBM X10 offer reified generics when compiled to Java (it can also compile to C++). Have a look at this: http://x10.sourceforge.net/documentation/papers/X10Workshop2012/slides/Takeuchi.pdf – Richard Gomes Aug 25 '12 at 09:54
  • 4
    This answer is misleading - there is no fundamental JVM limitation that prevents you from implementing reified generics. All reified generics require is that the compiler can emit type-specialised classes as needed, which is perfectly possible on the JVM using current JVM bytecode. Choosing to implement generics in the style of Java (using type erasure) is mainly a language/compiler design decision (which Java took for reasons of maintaining backwards compatibility). – mikera Sep 13 '12 at 02:14
  • The JVM bytecode has nothing to do with reified generics. This sort of thing is a matter of language specification, compiler features and runtime library support. IBM/X10 supports reified generics and it compiles to the JVM, demonstrating that ... yes... it can be done. – Richard Gomes Apr 25 '18 at 00:59
5

Just to complement oxbow_lakes, there's a question on Stack Overflow about how to get around type erasure in Scala.

Community
  • 1
  • 1
Daniel C. Sobral
  • 295,120
  • 86
  • 501
  • 681
3

"implicit Manifest" is a Scala compiler trick and it does not make generics in Scala reified. The Scala compiler, when it sees a function with "implicit m: Manifest[A]" parameter and it knows the generic type of A at the call site, it will wrap the class of A and its generic type parameters into a Manifest and make it available inside the function. However, if it could not figure out the true type of A, then it has no way of creating a Manifest. In other words, Manifest has to be passed along the function calling chain if the inner function needs it.

scala> def typeName[A](a: A)(implicit m: reflect.Manifest[A]) = m.toString
typeName: [A](a: A)(implicit m: scala.reflect.Manifest[A])java.lang.String

scala> typeName(List(1))
res6: java.lang.String = scala.collection.immutable.List[int]

scala> def foo[A](a: A) = typeName(a)
<console>:5: error: could not find implicit value for parameter m:scala.reflect.Manifest[A].
       def foo[A](a: A) = typeName(a)
                                  ^

scala> def foo[A](a: A)(implicit m: reflect.Manifest[A]) = typeName(a)
foo: [A](a: A)(implicit m: scala.reflect.Manifest[A])java.lang.String

scala> foo(Set("hello"))
res8: java.lang.String = scala.collection.immutable.Set[java.lang.String]
Walter Chang
  • 11,547
  • 2
  • 47
  • 36
  • Why don't they just invisibly add a Manifest to every method call that has generics? Do Manifests incur a performance penalty? – cdmckay Sep 01 '09 at 05:43
  • I'm not sure that anyone here has said that manifests make generics reified (which seems to be your implication). I said that Manifests can be imported to derive type information at runtime whereas reified generics would require a change to the bytecode specification. – oxbow_lakes Sep 01 '09 at 06:39
  • @oxbow_lakes I am sorry if my answer came across like that; I didn't mean to imply anyone said anything about "manifests make generics reified". All I wanted to point out is some limitations on using Manifest. – Walter Chang Sep 01 '09 at 08:19
3

To complement oxbow_lakes answer: It is no possible and it seems it will never happen (at least soon).

The (refutable) reasons JVM will not support reified generics seems to be:

  • Lower performance.
  • It breaks backward compatibility. It can be solved duplicating and fixing a lot of libraries.
  • It can be implemented using manifests: The "solution" and the biggest impediment.

References:

You can easily benchmark it and see that performance impact is very noticeable. Especially memory consumption increases a lot.

I believe the way to go is to have optional reification the way we start to do in Scala with Manifests/TypeTags.

If you can and combine it with runtime specialization you can aim for high performance and generic code. However, that's probably goal for Scala 2.12 or 2.13.

santiajo
  • 31
  • 4
1

Once scalac is a compiler, it has the potential of being able to embellish the generated code with whatever data structures are needed to implement reified generics.

What I mean is that scalac would have the ability to see...

// definition
class Klass[T] {
  value : T
}

//calls
floats  = Klass[float]
doubles = Klass[double]

... and "expand" to something like this:

// definition
class Klass_float {
  value : float
}
class Klass_double {
  value : double
}

// calls
floats  = Klass_float
doubles = Klass_double

Edit

The point is: the compiler has the ability to create all necessary data structures which demonstrate to be necessary to provide additional type information at runtime. Once this type information is available, the Scala runtime would take advantage of it and could perform all type-aware operations we can imagine. It does not matter whether the JVM provides bytecode for reified generics or not. The work is not done by the JVM, but by the Scala library.

If you have already written a symbolic debugger (I did!), you know that you can basically 'dump' all information the compiler has at compile-time into the generated binary, adopting whatever data organization demonstrates to be more convenient for further processing. This is exactly the same idea: 'dump' all type information the Scala compiler has.

In a nutshell, I don't see why it could not be possible, does not matter whether the JVM provides native operations for reified generics or not. The JVM bytecode has nothing to do with reified generics. This sort of thing is a matter of language specification, compiler features and runtime library support.

Another edit

IBM X10 demonstrates the ability I'm talking of: it compiles X10 code onto Java code, leveraging reified generics onto Java platforms. As I mentioned before: it can be done (and IBM X10 did!) but this kind of feature involves language specification, compiler support (or compiler plugins) and enough support in runtime libraries. More info at: http://x10.sourceforge.net/documentation/papers/X10Workshop2012/slides/Takeuchi.pdf

Richard Gomes
  • 5,675
  • 2
  • 44
  • 50
  • This has two problems. First, you multiply your code size. Second, you do not know all classes you have to generate at compile time. Take, for instance, a collection implementation from the library which you typically/potentially only have in compiled form. You would expect it to work with your new class `RichardsFoo`, wouldn'n you? – Raphael Feb 01 '11 at 08:22
  • 1
    This is easily bypassed by storing _erased_ compiled class. So when scalac encounters Klass[SomethingStrange] it gets the type-erased Klass and instantiates Klass_SomethingStrange. That's how .NET works at runtime, BTW. – Cyberax Apr 29 '11 at 02:46
  • 1
    @Raphael : The fact it multiplies code size is a problem only if you consider the cost/benefit bad. Second, please have a look at http://fastutil.di.unimi.it/. You see an example were specialized implementations exist on top of a generic API. The problem here is the multiplication of code size. As I said before, it may or may not be a problem. – Richard Gomes Jun 30 '12 at 15:11