NoSuchMethodError when using Scala in R with rscala

Question

I'm having trouble using rscala package in R to call precompiled Scala code.

I have an Scala project running on version 2.11.12. I build a fat jar with sbt-assembly and make it available to rscala through rscala::scala("path-to-jar.jar")

But whenever I try to run any function/method/whatever from the package, I get the following error:

java.lang.reflect.InvocationTargetException
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
    at myPackage.myClass$.myMethod(myClass.scala:116)
    at $line6.$read$$iw$$iw$$iw$$iw$.$anonfun$res0$1(<console>:16)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.ddahl.rscala.server.Server.$anonfun$invoke$3(Server.scala:305)
    at org.ddahl.rscala.server.Server.wrap(Server.scala:47)
    at org.ddahl.rscala.server.Server.invoke(Server.scala:305)
    at org.ddahl.rscala.server.Server.run(Server.scala:400)
    at org.ddahl.rscala.server.Main$.delayedEndpoint$org$ddahl$rscala$server$Main$1(Main.scala:105)
    at org.ddahl.rscala.server.Main$delayedInit$body.apply(Main.scala:11)
    at scala.Function0.apply$mcV$sp(Function0.scala:39)
    at scala.Function0.apply$mcV$sp$(Function0.scala:39)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
    at scala.App.$anonfun$main$1$adapted(App.scala:80)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.App.main(App.scala:80)
    at scala.App.main$(App.scala:78)
    at org.ddahl.rscala.server.Main$.main(Main.scala:11)
    at org.ddahl.rscala.server.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:106)
    at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
    at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
    at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:132)
    at scala.reflect.internal.util.ScalaClassLoader.run(ScalaClassLoader.scala:106)
    at scala.reflect.internal.util.ScalaClassLoader.run$(ScalaClassLoader.scala:98)
    at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:132)
    at scala.tools.nsc.CommonRunner.run(ObjectRunner.scala:28)
    at scala.tools.nsc.CommonRunner.run$(ObjectRunner.scala:27)
    at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
    at scala.tools.nsc.CommonRunner.runAndCatch(ObjectRunner.scala:35)
    at scala.tools.nsc.CommonRunner.runAndCatch$(ObjectRunner.scala:34)
    at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
    at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:73)
    at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:92)
    at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:103)
    at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:108)
    at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Error in pop(details, transcompileInfo, envir) : Invocation error.

It does allow me to import its classes and objects, so it does seem to be in the classpath, but that's it.

My build.sbt includes crossScalaVersions := Seq("2.11.12", "2.12.7", "2.13.0-M5") as required in the package.

Any thoughts? It seems like a scala version error, since running rscala::scalaConfig() shows me it is using scalaFullVersion 2.12.8, but I can't seem to choose a scala installation, so I can't really expect this to be the problem.

EDIT:

After Jasper-M's suggestion, I set environment variable RSCALA_SCALA_MAJORVERSION=2.11 and reexecuted the config with rscala::scalaConfig(download='scala'). Now it returns scalaMajorVersion as 2.11.

Now, when I create an instance of the scala bridge with rscala::scala(), it returns a message (not an error) saying:

cat: /release: No such file or directory

If I run println() it compiles with no issues. However, if I instantiate the scala bridge with a precompiled fat JAR of my own and do ANYTHING (like println), it returns the following error:

error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/reflect/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/reflect/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading Predef, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/Predef.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)

I'm really clueless here. Any ideas?

Why are you targeting multiple Scala versions? The Spark installation only has one. It's either 2.11 (Spark <2.4) or 2.12 (Spark >=2.4). What do you mean by "required by the package"? — stefanobaghino, Apr 30 '19 at 11:30
I'm not really, but the documentation suggests compiling with crossScalaVersion, probably because of this compatibility issue. — Edgar, May 03 '19 at 14:12
https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops — Dmytro Mitin, Apr 07 '23 at 05:08

score 5 · Accepted Answer · answered Apr 30 '19 at 13:04

5

It's most definitely a Scala version error.

My best guess is that you have to set the environment variable RSCALA_SCALA_MAJORVERSION=2.11 . E.g. by calling

Sys.setenv(RSCALA_SCALA_MAJORVERSION = "2.11")

and then run

rscala::scalaConfig(download = "scala")

answered Apr 30 '19 at 13:04

Jasper-M

14,966
2
26
37

Thanks. I'll try it and check if it works out. It does make sense. – Edgar May 03 '19 at 14:11
I think we are close to make it work. `scalaConfig()` now returns `scalaMajorVersion` as `2.11`, which is great. I previously tried changing the file directly, but it didn't work. Setting the environment variable worked as you said. Now, however, when I try to create the bridge with `rscala::scala()`, it prints `cat: /release: No such file or directory` but still works. If I import my JAR, however, it breaks saying `error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/reflect/package.class)' is broken` and other stuff. Ideas? I'll edit the post. – Edgar May 03 '19 at 20:16
1

Fun fact. I tried to make an import and it returned a huge log with a fun remark at the end: `Error in rbyte(socketIn) : Scala seems to have died.` – Edgar May 03 '19 at 20:32
@Edgar - I saw this same error today when trying to serialize an unsupported type (a `dttm` in my case). Switched it to a `String` with `format("%Y-%m-%DT%H-%m-%s")` and R happily sent it across. – Joe McMahon Aug 17 '22 at 01:27

NoSuchMethodError when using Scala in R with rscala

1 Answers1