4

I created a Serializable version of Guava's ImmutableRangeMap and Builder in Scala in order to use in my Spark application. I have a zero argument constructor in my SerializableImmutableRangeMap as well, so why do I get InvalidClassException: no valid constructor when I run my Spark application?

Here is my SerializableImmutableRangeClass object and class:

object SerializableImmutableRangeMap extends Serializable {
  final class SerializableBuilder[K <: Comparable[_], V]() extends Serializable {
    val keyRanges: RangeSet[K] = TreeRangeSet.create()
    val rangeMap: RangeMap[K, V] = TreeRangeMap.create()

    def put(range: Range[K], value: V): SerializableBuilder[K, V] = {
      checkNotNull(range)
      checkNotNull(value)
      checkArgument(!range.isEmpty(), "Range must not be empty, but was %s", range)
      if (!keyRanges.complement().encloses(range)) {
        // it's an error case; we can afford an expensive lookup
        for (entry: Entry[Range[K], V] <- JavaConversions.asScalaSet(rangeMap.asMapOfRanges().entrySet())) {
          val key: Range[K] = entry.getKey()
          if (key.isConnected(range) && !key.intersection(range).isEmpty()) {
            throw new IllegalArgumentException(
                "Overlapping ranges: range " + range + " overlaps with entry " + entry)
          }
        }
      }
      keyRanges.add(range)
      rangeMap.put(range, value)
      this
    }

    def putAll(rangeMap: RangeMap[K, _ <: V]): SerializableBuilder[K, V] = {
      for (entry <- JavaConversions.asScalaSet(rangeMap.asMapOfRanges().entrySet())) {
        put(entry.getKey(), entry.getValue())
      }
      this
    }

    def build(): SerializableImmutableRangeMap[K, V] ={
      val map: java.util.Map[Range[K], V] = rangeMap.asMapOfRanges()
      val rangesBuilder: ImmutableList.Builder[Range[K]] = new ImmutableList.Builder[Range[K]](map.size())
      val valuesBuilder: ImmutableList.Builder[V] = new ImmutableList.Builder[V](map.size())
      for (entry: Entry[Range[K], V] <- JavaConversions.asScalaSet(map.entrySet())) {
        rangesBuilder.add(entry.getKey())
        valuesBuilder.add(entry.getValue())
      }
      return new SerializableImmutableRangeMap[K, V](rangesBuilder.build(), valuesBuilder.build())
    }
  }

def builder[K <: Comparable[_], V](): SerializableBuilder[K, V] = {
    new SerializableBuilder[K, V]()
  }
}

class SerializableImmutableRangeMap[K <: Comparable[_], V](ranges: ImmutableList[Range[K]], values: ImmutableList[V]) extends ImmutableRangeMap[K, V](ranges, values) with Serializable {
  def this() {
    this(ImmutableList.of(), ImmutableList.of())
  }
}

And the stack trace:

java.io.InvalidClassException: com.google.common.collect.SerializableImmutableRangeMap; no valid constructor
        at java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:150)
        at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:768)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
        at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:94)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:60)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:64)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Corey Wu
  • 1,209
  • 1
  • 22
  • 39
  • I'd recommend deleting the whole `SerializableImmutableRangeMap` object and most of the stack trace. They don't add anything to the question, just make it longer. Also you talk about `SerializableImmutableRangeClass` but there is no such thing in the code. – Daniel Darabos Aug 25 '15 at 20:50

2 Answers2

12

The relevant bit of the documentation for Serializable:

To allow subtypes of non-serializable classes to be serialized, the subtype may assume responsibility for saving and restoring the state of the supertype's public, protected, and (if accessible) package fields. The subtype may assume this responsibility only if the class it extends has an accessible no-arg constructor to initialize the class's state. It is an error to declare a class Serializable if this is not the case. The error will be detected at runtime.

Your superclass (ImmutableRangeMap) does not have a no-arg constructor and is not Serializable. This is the problem. There is no requirement to have a no-arg constructor for Serializable classes (such as SerializableImmutableRangeMap).

Daniel Darabos
  • 26,991
  • 10
  • 102
  • 114
  • So is there any way to serialize `ImmutableRangeMap`? – Corey Wu Aug 25 '15 at 21:36
  • You could add a new class between `SerializableImmutableRangeMap` and `ImmutableRangeMap` in the inheritance chain that has a no-arg constructor and is _not_ `Serializable`. But what's the point? Deserialization will go through the no-arg constructor and you basically lose all data. This would already be a problem with your current code if it compiled. – Daniel Darabos Aug 25 '15 at 23:52
1

Long answer short, If you are facing this in Scala. If child class is serializable make parent class also serializable. I was facing same issue, resolved it by making parent serializable.

Ankit Kaplish
  • 15
  • 6