2

I have this error in my code (wordCount from Kafka) compiled with SBT 

[error] /home/hduser/sbt_project/project1/src/main/scala/sparkKafka.scala:4:35:            object kafka is not a member of package org.apache.spark.streaming`
[error] import org.apache.spark.streaming.kafka.KafkaUtils
 not found: value KafkaUtils
[error] val lines = KafkaUtils.createStream(ssc, "localhost:2181", "spark-stream           ing-consumer-group", Map("customer" -> 2))

The file build.sbt contains these dependencies:

libraryDependencies += "org.apache.spark" % "spark-core_2.12" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.12" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.12" % "2.4.0"

How can I correctly import KafkaUtils?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
andrea5
  • 77
  • 2
  • 11

1 Answers1

4

KafkaUtils is in the org.apache.spark.streaming.kafka010 package (note that the namespace for the import includes the version kafka010).

From Spark Streaming Kafka Documentation:

import org.apache.spark.streaming.kafka010._
// ...

// val streamingContext = ...

val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "localhost:9092",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "group.id" -> "use_a_separate_group_id_for_each_stream",
  "auto.offset.reset" -> "latest",
  "enable.auto.commit" -> (false: java.lang.Boolean)
)

val topics = Array("topicA", "topicB")
val stream = KafkaUtils.createDirectStream[String, String](
  streamingContext,
  PreferConsistent,
  Subscribe[String, String](topics, kafkaParams)
)

Note: It's usually recommended to use Spark Structured Streaming with Kafka instead.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks. But in the line `val stream = KafkaUtils.createDirectStream[String, String]` What I insert in place of **String**? `val lines= KafkaUtils.createDirectStream[000.000.000.000:2181, String]` In this way I have 2 error – andrea5 Jan 13 '19 at 14:42
  • Nothing... Those are the key and value types, which correspond to the `classOf[StringDeserializer]`... You get errors because you changed it. If your topic has strings for keys and values, then this code works as-is. The only thing you need is the `streamingContext`. And again, do not use `:2181` to connect to Kafka – OneCricketeer Jan 13 '19 at 17:39
  • Thanks but I still have error: `......[error] cannot be applied to (String, scala.collection.immutable.Map[String,Object]) [error] Subscribe[String, String](topics, kafkaParams) [error] ^ [error] one error found [error] (Compile / compileIncremental) Compilation failed [error] Total time: 2 s, completed Jan 14, 2019 1:30:35 PM` I copied the code from [link](https://spark.apache.org/docs/2.2.0/streaming-kafka-0-10-integration.html) – andrea5 Jan 14 '19 at 13:31
  • Sorry, I cannot tell what the `^` arrow is pointing at, but I assume there is something you've written before that line, that is causing the problem. Please edit the initial question to format the full error, and your new code. However, I think it would be better to post a new question, because I think we've solved your KafkaUtils class not being found. – OneCricketeer Jan 14 '19 at 15:48