No such method running forEach in Scala job on Apache Spark

Question

I'm running a very simple Scala job on Apache Spark 2.4.5 and when I try and iterate over the columns in a DataFrame and print there names I get the following stack trace correlating to the line where I try and call the for each.

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
at SimpleApp$.main(SimpleApp.scala:10)
at SimpleApp.main(SimpleApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I am running Apache Spark in Docker using this image: bde2020/spark-master:2.4.5-hadoop2.7 I am compiling my app using scalaVersion := "2.12.11"

Full application code is:

import org.apache.spark.sql.{Row, SparkSession}

object SimpleApp {
  def main(args: Array[String]) {
    val file = "/spark/jobs/job1/data/test.json"
    val spark = SparkSession.builder.appName("Simple Application Scala").getOrCreate()
    val testData = spark.read.json(file)
    println("prints fine")
    testData.columns.foreach(x => println(x))
    spark.stop()
  }

build.sbt file is

name := "spark-scala"

version := "0.1"

scalaVersion := "2.12.11"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.5"

I am at a loss, I have checked and checked I am running the correct versions of things but suspect I must of missed something!

could try adding `spark-core` dependency.. ` libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.5" ` — notNull, Apr 13 '20 at 18:56
https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops — Dmytro Mitin, Apr 07 '23 at 05:04

score 1 · Accepted Answer · answered Apr 13 '20 at 19:43

1

After much head banging discovered the image actually uses Scala 2.11.12 which is deprecated with Spark 2.4.5! Obvious in hindsight, all working now.

answered Apr 13 '20 at 19:43

Sutty1000

767
1
14
32

score 0 · Answer 2 · answered Apr 13 '20 at 19:27

0

You are not setting spark-core in your dependencies.

answered Apr 13 '20 at 19:27

Ygor de Fraga

51
9

No such method running forEach in Scala job on Apache Spark

2 Answers2