0

We are using Spark 3.3.2 with Scala 2.12.15 and JDK 11.0.16

I don't have any scala code. Purely python and RDD.

Still we are getting the below error in rancher.

java.io.InvalidClassException: org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$RetrieveSparkAppConfig$; local class incompatible: stream classdesc serialVersionUID = .....

I have searched so many topics in Stack overflow. But everywhere they are telling version incompatability. We have all versions up to date and we can't modify..

Please suggest some proper solution. Thanks in advance

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
chukka
  • 11
  • 7
  • Is `org.apaches` a typo? Should it be `org.apache`? *"I don't have any scala code"* Are you using PySpark? Then you do have Scala code. The class `org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveSparkAppConfig` is from [spark-core](https://mvnrepository.com/artifact/org.apache.spark/spark-core). Just in case, hardly this is the reason but did you try to run with JDK 8 rather than 11? – Dmytro Mitin Apr 11 '23 at 08:21
  • The [question](https://stackoverflow.com/questions/10378855/java-io-invalidclassexception-local-class-incompatible) says that most probably `java.io.InvalidClassException: local class incompatible: stream classdesc` is because incompatible versions of a dependency are used (with different `serialVersionUID`). – Dmytro Mitin Apr 11 '23 at 08:22
  • Let's look at your Java classpath. Can you run the following Scala code inside your PySpark environment? https://scastie.scala-lang.org/DmytroMitin/KfLOLvdmSLqOL6rKfnKFow or at least https://scastie.scala-lang.org/DmytroMitin/KfLOLvdmSLqOL6rKfnKFow/1 This is how to run Scala code in PySpark: https://aseigneurin.github.io/2016/09/01/spark-calling-scala-code-from-pyspark.html https://stackoverflow.com/questions/36023860/how-to-use-a-scala-class-inside-pyspark Why do you say that you can't modify the versions? It's possible that you'll have to. – Dmytro Mitin Apr 11 '23 at 08:22
  • *"We have all versions up to date"* Why do you consider Scala 2.12.15 up to date? https://mvnrepository.com/artifact/org.scala-lang/scala-library – Dmytro Mitin Apr 11 '23 at 09:40
  • @DmytroMitin Thanks for the reply and it's a typo (apaches). We have this spark 3.3.2 image to use in our organization and we don't have any environment to edit the versions of JDK and Scala. We have access to use the image of spark – chukka Apr 11 '23 at 12:12
  • Is it possible that you need another image? Is your image working? I mean do you not always get `InvalidClassException`, is it sometimes working? Would you show your classpath? – Dmytro Mitin Apr 11 '23 at 12:30
  • What do you mean by Spark image? Is it Docker image, [image data source](https://spark.apache.org/docs/latest/ml-datasource.html#image-data-source), something else? – Dmytro Mitin Apr 12 '23 at 08:40
  • 1
    Yes it's docker image. We are just using it without modifying the versions. – chukka Apr 12 '23 at 12:09

0 Answers0