Issue when running the job in EMR 6.6.0 using Spark 3.2.0

Question

My job(Spark-Scala) is running fine in emr-5.36.0 using spark 2.4.8 and scala 2.11.

But I am getting below error while running the same job in emr-6.6.0 using spark 3.2.0 and scala 2.12.15

Exception in thread "main" java.lang.IncompatibleClassChangeError: Class org.json.JSONArray does not implement the requested interface java.lang.Iterable

As per my understanding below part of code is causing issue

*import scala.collection.JavaConverters._
val configJSON: JSONArray = fetchConfigs(configServerUri, configFiles)
configJSON.asScala.foreach(propertySource => {
  propertySource.asInstanceOf[JSONObject].getJSONObject("source").toMap.asScala.foreach(kv => {
    sparkAppConf.set(configReMap.getOrElse(kv._1, kv._1), kv._2.toString)
  })*

I am using below dependency in pom for json:

<dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20220924</version>
</dependency>

And JSONArray is implementing the Iterable:

**public class JSONArray implements Iterable<Object>**

Note:The same code is running fine in local with spark 3.2.0 and scala 2.12.15.

Any chance EMR already provides another version of the org.json library? — Gaël J, Dec 27 '22 at 20:53
https://stackoverflow.com/questions/1980452/what-causes-java-lang-incompatibleclasschangeerror *"It's generally the result of two mildly different versions of the same dependency on the classpath..."* *"try launching the VM with `-verbose` as an argument, then look at the classes that were being loaded when the exception occurs. You should see some surprising information. For instance, having multiple copies of the same dependency..."* — Dmytro Mitin, Dec 28 '22 at 12:19

Issue when running the job in EMR 6.6.0 using Spark 3.2.0

0 Answers0