1

I have exported two jars of two apps I have in the same Kafka/Spark Streaming project. The jar with the Kafka Producer works fine. The jar with the Spark Consumer returns this error:

NoClassDefFoundError: kafka/serializer/StringDecoder

They share the same dependencies folder, which I have obviously exported in order to make the jars work outside Eclipse.

In Eclipse they both work fine.

How can I fix this?

sirdan
  • 1,018
  • 2
  • 13
  • 34
  • do you kafka jar in your class path ? – Gaurav Jun 15 '17 at 09:00
  • I've got spark-streaming-kafka in the classpath. Isn't it enough? – sirdan Jun 15 '17 at 09:03
  • but stringDecoder is part of kafka module just check whether kafka-streaming contains String decoder – Gaurav Jun 15 '17 at 09:13
  • I looked into the jar of spark-streaming-kafka and it contains the .class file that is named in the error: `org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:55)`. So I can't understand why he misses this class. – sirdan Jun 15 '17 at 09:18

1 Answers1

2

You need Kafka itself in the class path, not just spark-streaming-kafka. In Eclipse you do, because your build system retrieves all dependencies, and Eclipse integrates with it.

I looked into the jar of spark-streaming-kafka and it contains the .class file that is named in the error: org.apache.spark.streaming.kafka.KafkaUtils$.createStream(Ka‌​fkaUtils.scala:55). So I can't understand why he misses this class.

Presumably you mean that contains org.apache.spark.streaming.kafka.KafkaUtils$, but when JVM tries to load it, it discovers that it needs other classes, including kafka.serializer.StringDecoder, which is not there.

Alexey Romanov
  • 167,066
  • 35
  • 309
  • 487
  • Thanks for helping. So, which jars you think I need to import to fix this? Because I've been having troubles with incompatible versions of Spark and Kafka. – sirdan Jun 18 '17 at 08:35
  • You really don't provide enough information to answer this. But see https://stackoverflow.com/questions/275120/java-how-do-i-know-which-jar-file-to-use-given-a-class-name. – Alexey Romanov Jun 18 '17 at 08:47
  • Solved. I missed all the Kafka jars, which I imported only with Maven dependencies, so Eclipse couldn't actually export them. Your hint solved the issue, so thank you so much and well deserved bounty. – sirdan Jun 18 '17 at 09:03