0

I have the following spark code stated below

spark-submit --class lr --master yarn --deploy-mode client --driver-memory 4g  --executor-memory  --num-executors 4 --executor-cores 4 --queue xxxxx_p1  /dd/ff/ff/fff/dd/dd/ddd/ddd/Automation/full-1.0-SNAPSHOT.jar  /dd/dd/dd/dd/dd/dd/dd/dd/dd/lr_srvc_dt.properties

When I run this I get the following error,

22/06/29 04:50:41 ERROR SparkSubmit$$anon$2: 'scala.collection.mutable.ArrayOps scala.Predef$.refArrayOps(java.lang.Object[])'
java.lang.NoSuchMethodError: 'scala.collection.mutable.ArrayOps scala.Predef$.refArrayOps(java.lang.Object[])'
        at labresult$.main(labresult.scala:30)
        at labresult.main(labresult.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1060)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1069)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Build definitions:

Scala version 2.12.15, OpenJDK 64-Bit Server VM, 11.0.14.1

Spark_Version:- version 3.2.0.3-eep-810

JDK-8

I can't figure out what causes this. And how to resolve such issue? Any ideas?

Palash Chaturvedi
  • 1
  • 1
    This is likely cause by either Scala standard library not in the classpath or a conflict of version between the Scala version used at compile time versus the one at runtime. – Gaël J Jun 29 '22 at 16:57
  • https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops – Dmytro Mitin Apr 07 '23 at 04:45

0 Answers0