2

Which Scala version works with Spark 2.2.0 ? I'm getting following error:

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

Oleg Svechkarenko
  • 2,508
  • 25
  • 30
Hussain Asghar
  • 35
  • 1
  • 6
  • Is that happening with scala test ? – Avishek Bhattacharya Sep 29 '17 at 17:44
  • I got this error fixed and now came up with a new one.The error was removed by adding dependency in build.sbt – Hussain Asghar Sep 29 '17 at 18:16
  • I'm still getting the error, I think it's version conflict, but tried everything and it still didn't worked. – Hussain Asghar Sep 29 '17 at 18:46
  • similar question [here](https://stackoverflow.com/questions/43192915/exception-in-thread-main-java-lang-nosuchmethoderror-scala-predef-refarrayop) – geo Sep 29 '17 at 21:07
  • name := "Scala-Spark" version := "1.0" scalaVersion := "2.11.8" // https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0" copy & paste this in build.sbt. It will definately work. – Hussain Asghar Sep 29 '17 at 21:42
  • https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops – Dmytro Mitin Apr 07 '23 at 04:50

4 Answers4

3

From the documentation here:

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.2.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).

Joe K
  • 18,204
  • 2
  • 36
  • 58
1

Spark 2.2.0 is built and distributed to work with Scala 2.11 by default. To write applications in Scala, you will need to use a compatible Scala version (e.g. 2.11.X). And your scala version might be 2.12.X. That's why it is throwing exception.

Mahesh Chand
  • 3,158
  • 19
  • 37
1

To select appropriate scala version for your spark application one could run spark-shell on the target server. Desired scala version is contained in the welcome message:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0.2.6.3.0-235
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)

It is 2.11.8 in my Spark distributive.

Also there are pages on MVN repository contained scala version for one's spark distribution:

https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11

https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12

Oleg Svechkarenko
  • 2,508
  • 25
  • 30
0

Spark 2.2.0 needs Java 8+ and scala 2.11. Thats about the version info.

But, looking at your error "Exception in thread "main" java.lang.NoSuchMethodError: ", it seems your Spark is unable to find the driver class.

Probably you should be looking in this direction rather than versions.

Abhay Dandekar
  • 1,230
  • 10
  • 30