1

I have a fairly complex object graph serialized out from Scala-2.9 and I need to read it into Scala-2.10. However, somewhere deep in the object graph Scala-2.10 throws:

! java.lang.ClassNotFoundException: scala.collection.JavaConversions$SeqWrapper
! at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_21]
! at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_21]
! at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_21]
! at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_21]
! at java.lang.ClassLoader.loadClass(ClassLoader.java:423) ~[na:1.7.0_21]
! at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_21]
! at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ~[na:1.7.0_21]
! at java.lang.Class.forName0(Native Method) ~[na:1.7.0_21]
! at java.lang.Class.forName(Class.java:266) ~[na:1.7.0_21]
! at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:623) ~[na:1.7.0_21]
...

What is the simplest way to load this serialized object into Scala-2.10? The object deserializes correctly with Scala-2.9, but looks like things have moved around in the standard library. Most of the members of scala.collection.JavaConversions are now in scala.collection.convert.Wrappers

Going forward, I am also interested in more robust ways of persisting large complex object graphs without having to explicitly specify the serialization for every class involved.

Daniel Mahler
  • 7,653
  • 5
  • 51
  • 90
  • 1
    https://speakerdeck.com/heathermiller/on-pickles-and-spores-improving-support-for-distributed-programming-in-scala may be of interest. – som-snytt Jun 22 '13 at 06:32
  • thanks @som-snytt. I saw Heather's talk at ScalaDays. Scala-pickling looks great, but afaik it is not ready yet production use. – Daniel Mahler Jun 22 '13 at 21:06
  • What version of Java? Presumably the same for both sides. The SeqWrapper is just the wrapper for java collections. I wouldn't speculate about the CNFE without ruling out the obvious classpath problems. – som-snytt Jun 27 '13 at 05:51
  • Scratch the comment about SeqWrapper -- someone asked about 2.9 and I was still on that branch -- and I have to watch hulu's 101 days of summer for Master of the House. – som-snytt Jun 27 '13 at 06:08

4 Answers4

2

Thoughts, of no real help:

  1. You've bumped up against a change in underlying implementation for Scala collections that is reflected in the serialization of same. You're unable to "just load it" in to 2.10, so you need some common ground.

  2. You're likely to run into this with every version of Scala, as the Collections haven't fully settled.

  3. I presume your intent is to load your graph using 2.9 based code, convert to some new format, and dump in a new "common" format.

  4. In the java world, I'd reach for JAXB or SDO; perhaps EclipseLink MOXy. I doubt MOXy will be aware of Scala collection types.

  5. I presume you've already seen this.

  6. Can your object graph be converted to something based entirely on core Java data types?

Community
  • 1
  • 1
Richard Sitze
  • 8,262
  • 3
  • 36
  • 48
  • thanks @RichardSitze. I suspect I will have to reload in 2.9 & reserialize to something bot 2.9 & 2.10 can understand. However, I am dealing with fairly complicated application objects, so I am looking for the most painless solution, ie not having to explicitly spell out how to serialize & deserialize every class & field involved. I also still have some hope that I could use the new 2.10 reflection apis to intercept the SeqWrapper deserialization directly in 2.10, but that may not be possible. – Daniel Mahler Jun 24 '13 at 20:02
1

Please don't downvote as a knee-jerk, but my thought was to deserialize in one class loader (on scala 2.9), convert to a java collection, then in a second class loader (with 2.10 on the class path) convert from java back to scala.

In other words, java is the common format (the java runtime is common to both class loaders).

(Alternatively, instead of two classloaders, try serializing the java forms, then scarfing it back.)

I think this goes to Richard's #6. It doesn't have to be entirely java core, only what is compatible.

I'll try to come up with an example during a coffee break, but of course the caveat would be that the incompatibility rests with the collection and not with what is collected.

som-snytt
  • 39,429
  • 2
  • 47
  • 129
  • Thanks!! I had a complete blind spot for using a`ClassLoader` to solve this problem. I think I can actually do something simpler: just write one loader that intercept the loading of the offending classes & redirects to their 2.10 equivalents & smooth over the `SerialVersionUID` differences. This way I should be able to read the original file directly into 2.10 without re-serialization. This would be close to the my best case scenario solution. – Daniel Mahler Jun 28 '13 at 18:50
  • @DanielMahler For the specific issue of collections, I had imagined producing java collections, but that is not a built-in facility; JavaConverters are wrappers; but you already have a wrapped java collection, so it might be easier to just unwrap it. I was thinking no special knowledge about the classes. – som-snytt Jun 28 '13 at 19:05
  • The suggestions to use ClassLoaders lead me to a working solution. My actual solution is different (afaict the suggested approach would work too), but this answer was the lightbulb I needed. Thanks!! – Daniel Mahler Jun 29 '13 at 20:28
  • OK, I'm so pleased for you, @DanielMahler! I haven't looked it up yet, but the past tense "led" is spelled like "fed" and not "read" (pronounced like "red") or the metal lead. If I could pick a superpower, it would not be flight but the ability to map { case "lead" if past => "led" } instantaneously. As a child, I pronounced "misled", as I read to myself, as "missled". Maybe that's why. – som-snytt Jun 29 '13 at 20:41
  • @DanielMahler BTW about a decade ago, I did a similar thing with a project that upgraded its product version and broke serialized data for its installed base. I had an intermediate language to describe how to transform the serialized form. – som-snytt Jun 29 '13 at 20:49
0

This will probably blow up in your face, but you could try putting the 2.9 scala-library.jar on your classpath after the 2.10 scala-library.jar. The class loader should then find

 scala.collection.JavaConversions$SeqWrapper

, but I'd be surprised if the serialization goes all the way through ...

You should consider just serializing to json with gson or jackson or lift-json or whatever. It's tedious, but you do it once, and it's done, and it will work with other languages too - your "quick" solution will mean recurring pain going forward ...

Good luck!

Reuben

Reuben
  • 155
  • 3
0

So here is what ended up working for me, thanks to @som-snytt for pointing me in the right direction:

object MyWrappers {
  import java.{ lang => jl, util => ju }, java.util.{ concurrent => juc }
  import scala.collection.convert._
  import WrapAsScala._
  import WrapAsJava._
  import Wrapper._

  @SerialVersionUID(3200663006510408715L)
  case class SeqWrapper[A](underlying: Seq[A]) extends ju.AbstractList[A] with Wrappers.IterableWrapperTrait[A] {
    def get(i: Int) = underlying(i)
  }
}

import org.apache.commons.io.input.ClassLoaderObjectInputStream
object Loader extends ClassLoader {
  override def loadClass(name: String) : Class[_] = {
    import javassist._
    try super.loadClass(name)
    catch {
      case e: ClassNotFoundException if name.startsWith("scala.collection.JavaConversions") => {
            val name2 = name.replaceFirst("scala.collection.JavaConversions",
                                          "MyWrappers")
            val cls = ClassPool.getDefault().getAndRename(name2, name)
            cls.toClass()
          }
    }
  }
}

val objectStream = new ClassLoaderObjectInputStream(Loader, stream)
objectStream.readObject()

This allows me to read in my original 2.9 serialized files directly into 2.10 without re-serialization. It depends on Javassist to do the Class indirection and uses ClassLoaderObjectStream from Apache Commons, even though that would be simple to roll your own. I am not that happy that I had to make my own copy of SeqWrapper (this turned out to be the only offending class in my file), but the wrapper classes in scala-2.10's scala.collection.convert.Wrappers have different SerialVersionUIDs than the corresponding classes in 2.9's scala.collection.JavaConversions even though the sources are textually identical. I originally tried to just redirect to scala.collection.convert.Wrappers and set the SerialVersionUID with Javassist:

object Loader extends ClassLoader {
  override def loadClass(name: String) : Class[_] = {
    import javassist._
    try super.loadClass(name)
    catch {
      case e: ClassNotFoundException if name.startsWith("scala.collection.JavaConversions") => {
            val name2 = name.replaceFirst("JavaConversions", "convert.Wrappers")
            val cls = ClassPool.getDefault().getAndRename(name2, name)
            cls.addField(CtField.make("private static final long serialVersionUID = 3200663006510408715L;", cls))
            cls.toClass()
          }
    }
  }
}

That allowed me to read the serialized file without exception, but the object that was read in that way was incomplete. (If this approach worked and there was more than one problem class in the file, I would really need a lookup table for the SerialVersionUIDs, but that is beside the point). If anybody knows a way set the SerialVersionUID on the Javassist generated class without breaking anything else something else I would like to hear it.

Daniel Mahler
  • 7,653
  • 5
  • 51
  • 90