3

The problem is that every job fails with the following exception:

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
at ps.sparkapp.Classification$.main(Classification.scala:35)
at ps.sparkapp.Classification.main(Classification.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

This exception meas that the task can not find the method. I develop using the intelij community edition. I have no problems compiling the package. All dependencies are packaged correctly. Here my build.sbt:

name := "SparkApp"
version := "1.0"

scalaVersion := "2.11.6"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.1"


 scala -version 
 Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL

I found out that this error has somehow to do with scala because it only happens when I use functionality that is native to scala, e.g scala for loop, .map or .drop(2). The class and everything is still written in scala, but if i avoid functionality like .map or drop(2) then everything works fine.

import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.sql.SparkSession
import org.apache.spark.ml.linalg.Vector

object Classification {

  def main(args: Array[String]) {
    ...
    //df.printSchema()
    var dataset = df.groupBy("user_id","measurement_date").pivot("rank").min()

    val col = dataset.schema.fieldNames.drop(2) // <- here the error happens

    // take all features and put them into one vector
    val assembler = new VectorAssembler()
      .setInputCols(col)
      .setOutputCol("features")

    val data = assembler.transform(dataset)
    data.printSchema()
    data.show()

    sc.stop()
  }

}

As said if I do not use .drop(2) everything works perfectly, but avoiding these methods is no option since that is very painful..

I could not find any solution on the web, any ideas?

BTW: I can use these methods within the spark-shell, which i find strange.

Thanks in advance.

NOTE 1)

I use: SPARK version 2.1.1

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_131)

user4054919
  • 129
  • 13
  • 3
    This is usually the result of a Scala version mismatch; Your SBT file looks OK, but maybe you have a different Scala version on the master/worker machines you're connecting to? Are you using a local Spark session (master = `local`, `local[*]`...) or connecting to some pre-deployed Spark Standalone / YARN / Mesos master? – Tzach Zohar Jul 28 '17 at 15:53
  • I am using spark in standalone mode, master & slave are on the same machine. – user4054919 Jul 28 '17 at 15:56
  • Can you check what Scala version these processes (master / slave) use? – Tzach Zohar Jul 28 '17 at 16:00
  • I just checked, spark-shell notes: "Using Scala version 2.11.8" There is indeed a mismatch. – user4054919 Jul 28 '17 at 16:02
  • I updated the scala version, but the problem stays.. – user4054919 Jul 28 '17 at 17:20
  • 2
    Okay, @Tzach Zohar your solution was correct. I did not notice that sbt changed the output dir for the new jar and tested the old jar, my fault. Thanks for your quick awnser/solution! – user4054919 Jul 28 '17 at 17:29
  • https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops – Dmytro Mitin Apr 07 '23 at 05:07

1 Answers1

0

Try adding the actual Scala libraries etc as a project dependency. E.g.:

libraryDependencies += "org.scala-lang" % "scala-library" % "2.11.6"

nairbv
  • 4,045
  • 1
  • 24
  • 26