Are traits and interfaces binary compatible?

Question

I've just kind of surprised by the fact that Scala is binary incompatible across different releases. Now, since in Java 8 we have default method implementations which is pretty much the same as traits provide us with is it safe to use traits in Java code? I tried to use it myself as this:

trait TestTrait {
  def method(v : Int)
  def concrete(v : Int) = println(v)
}

public class Test implements TestTrait{ // Compile-error. Implement concrete(Int)
    @Override
    public void method(int v) {
        System.out.println(v);
    }
}

But it refuses to compile. Compiler complaints about not imlementing concrete(Int). Although I specified the implementation in TestTrait.

Answer 1

When the Scala 2.11 compiler compiles a trait, it doesn't generate an interface with default methods, because the generated code has to work with Java 6. In Scala 2.12 (which requires Java 8) it will, so if you compile your Scala code with 2.12 compiler, I expect that you should be able to use it from Java in this way (at least for a simple case like this).

Note that changes like this are precisely what makes different Scala versions binary incompatible: if you tried to use a trait compiled with Scala 2.11 from Scala 2.12, it would try to call interface's default methods, which aren't there.

Answer 2

You are having conflicting expectations.

You are "surprised" to see that Scala is binary-incompatible between major releases, which suggests that you expect the opposite: that Scala should be binary-compatible even between major releases.

And yet at the same time you expect Scala to use an encoding for traits that relies on a feature that didn't even exist when the binary format for Scala 2.11 was designed. The first Release Candidate for Scala 2.11, i.e. the point where no more changes are allowed, was two weeks before Java 8 was even released. Requiring every Scala user to have Java 8 installed before it is even released, would be ridiculous.

So, on the one hand, you expect full binary-compatibility, i.e. no changes at all. On the other hand, you expect to use the latest and greatest features, i.e. changes as fast as possible. You can't have both. You have to choose.

And, as Alexey already pointed out in his answer, it is precisely improvements like this, that require breaking binary compatibility.

If you have binary compatibility, you cannot change your binary representation if you figure out a better one. And you cannot make use of new features of the target platform when they become available. This is very restrictive, especially for a language like Scala which pushes the boundaries of what can be reasonably encoded on the JVM. It would be very harsh on the compiler designers to force them to get "everything right" the very first time.

Here are a couple things that have been changed over the years and broken backward compatibility:

• The encoding of lambdas, using MethodHandles, when they were added in Java 7. They couldn't have "gotten this right the first time", because back then MethodHandles didn't even exist.
• (In the upcoming 2.12.) The encoding of lambdas, again, so that they are identical to Java 8's encoding. They couldn't have "gotten this right the first time", because back then lambdas didn't even exist in Java.
• (In the upcoming 2.12.) The encoding of traits using default methods in interfaces. They couldn't have "gotten this right the first time", because back then default methods didn't even exist in Java.

Should the Java platform ever get Proper Tail Calls or at least Proper Tail Recursion, I'm pretty sure, the ABI will change again to make use of those features. And if we get Value Types in the JVM, the encoding of Value Classes in Scala is likely to change.

However, in dotc, the compiler for Dotty, the team is trying a new approach to binary compatiblity: TASTy. TASTy is a serialization format for Typed Abstract Syntax Trees. The idea is that binary compatibility is guaranteed for TASTy, but not for the final output. TASTy contains all necessary information to re-compile the program, so if you want to combine two closed-source libraries compiled by different compilers, that's not a problem, because you can just throw away the compiled code and recompile from TASTy.

The TASTy is going to be shipped together with the compiled code, always. E.g. for Scala-JVM, the serialized TASTy would be shipped in the metadata section of the .class file or .jar, for Scala.js in a comment or binary array inside the compiled source file, for Scala-native in the metadata section of the compiled .dll, .exe, .so, .dylib, and so forth.

Getting back to your specific question about traits:

At the moment, a single trait is encoded as:

• an interface containing abstract declarations for all the trait's methods (both abstract and concrete)
• a static class containing static methods for all the traits's concrete methods, taking an extra parameter $this
• at every point in the inheritance hierarchy where the trait is mixed in, synthetic forwarder methods for all the concrete methods in the trait that forward to the static methods of the static class

So, the following Scala code:

trait A {
  def foo(i: Int) = i + 1
  def abstractBar(i: Int): Int
}

trait B {
  def baz(i: Int) = i - 1
}

class C extends A with B {
  override def abstractBar(i: Int) = i * i
}

would be encoded like this:

interface A {
    int foo(int i);
    int abstractBar(int i);
}

abstract class A$class {
    static void $init$(A $this) {}
    static int foo(A $this, int i) { return i + 1; }
}

interface B {
    int baz(int i);
}

abstract class B$class {
    static void $init$(B $this) {}
    static int baz(B $this, int i) { return i - 1; }
}

class C implements A, B {
    public C() {
        A$class.$init$(this);
        B$class.$init$(this);
    }

    @Override public int baz(int i) { return B$class.baz(this, i); }
    @Override public int foo(int i) { return A$class.foo(this, i); }
    @Override public int abstractBar(int i) { return i * i; }
}

But in Scala 2.12 targeting Java 8, it looks more like this:

interface A {
    static void $init$(A $this) {}
    static int foo$(A $this, int i) { return i + 1; }
    default int foo(int i) { return A.foo$(this, i); };
    int abstractBar(int i);
}

interface B {
    static void $init$(B $this) {}
    static int baz$(B $this, int i) { return i - 1; }
    default int baz(int i) { return B.baz$(this, i); }
}

class C implements A, B {
    public C() {
        A.$init$(this);
        B.$init$(this);
    }

    @Override public int abstractBar(int i) { return i * i; }
}

As you can see, the old design with the static methods and forwarders has been retained, they are just folded into the interface. The trait's concrete methods have now been moved into the interface itself as static methods, the forwarder methods aren't synthesized in every class but defined once as default methods, and the static $init$ method (which represents the code in the trait body) has been moved into the interface as well, making the companion static class unnecessary.

It could probably be simplified like this:

interface A {
    static void $init$(A $this) {}
    default int foo(int i) { return i + 1; };
    int abstractBar(int i);
}

interface B {
    static void $init$(B $this) {}
    default int baz(int i) { return i - 1; }
}

class C implements A, B {
    public C() {
        A.$init$(this);
        B.$init$(this);
    }

    @Override public int abstractBar(int i) { return i * i; }
}

I'm not sure why this wasn't done. At first glance, the current encoding might give us a bit of forwards-compatibility: you can use traits compiled with a new compiler with classes compiled by an old compiler, those old classes will simply override the default forwarder methods they inherit from the interface with identical ones. Except, the forwarder methods will try to call the static methods on A$class and B$class which no longer exist.