Spark Streaming Kafka with Java11 scala code Issue

Question

I am using scala Spark Streaming Kafka with JDK 11. But I am getting the below error.

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

Below is the code I am using.

val conf = new SparkConf().setMaster("local[*]").setAppName("KafkaExample")
      .set("spark.mongodb.input.uri", "mongodb://127.0.0.1/db.table_data")
      .set("spark.mongodb.output.uri", "mongodb://127.0.0.1/db.table_data")
      .set("spark.driver.allowMultipleContexts", "false")
      .set("spark.ui.enabled", "false")

val kafkaParams = Map("metadata.broker.list" -> "localhost:9092")
    val topics = List("topic").toSet
    val lines = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
      ssc, kafkaParams, topics)

Below is my pom.xml

<dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_2.12</artifactId>
          <version>2.4.8</version><!--<version>2.3.2</version>-->
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-sql_2.12</artifactId>
          <version>2.4.8</version>
      </dependency>
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-streaming_2.12</artifactId>
          <version>2.4.8</version><!--<version>2.3.2</version>-->
          <!--<scope>provided</scope>-->
      </dependency>
      <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
      <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-streaming-kafka_2.11</artifactId>
          <version>1.6.1</version>
      </dependency>
     
      <dependency>
          <groupId>org.mongodb.spark</groupId>
          <artifactId>mongo-spark-connector_2.12</artifactId>
          <version>2.4.3</version>
      </dependency>
      <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
      <dependency>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>jackson-core</artifactId>
          <version>2.10.0</version>
      </dependency>

The issue is coming in KafkaUtils.createDirectStream() and i checked the maven and scala compatibility JDK 11 is compatible with scala 2.12 and higher and we don't have any maven dependency for spark-streaming-kafka jar. KIndly let me know if my analysis is wrong and which jar should i use for spark-streaming-kafka for JDK 11

spark-streaming-kafka_2.11 will not work with scala 2.12. 2.11 and 2.12 are not binary compatible. Maybe you need to use https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 which target's 2.12 [is your Kafka broker >0.10](https://spark.apache.org/docs/3.1.2/streaming-kafka-0-10-integration.html#content) ? — roby, Jun 07 '21 at 08:11
My kafka version is kafka_2.13-2.6.0 is it greater than 0.10. Also when i use the jar dependency you have suggested, It will require a lot of code change in KafkaUtils.createDirectStream() is it necessary? — m b, Jun 07 '21 at 09:44
Got help from here and got the issue resolved. https://stackoverflow.com/questions/55923943/how-to-fix-unsupported-class-file-major-version-55-while-executing-org-apache — m b, Jun 07 '21 at 10:35

score 0 · Answer 1 · answered Jun 07 '21 at 12:46

DirectStream API is deprecated, you should be using spark-sql-kafka-0-10 dependencies. The Mongo Spark driver works with Spark SQL also
As commented, you're mixing your Scala versions (the broker Scala version doesn't matter), and you're also mixing your Spark versions

Spark Streaming Kafka with Java11 scala code Issue

1 Answers1