Kafka with Spark throw Could not initialize class kafka.utils.Log4jController Error

Question

I am trying to write a kafka consumer in java using Apache spark. The code is not executing due to some Log4jController error. Don't know what I am missing.

pom.xml file is as follows:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.25</version>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.11</artifactId>
  <version>2.3.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.3.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.11</artifactId>
  <version>1.0.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Got following error

5645 [dag-scheduler-event-loop] INFO  org.apache.spark.scheduler.DAGScheduler  - ResultStage 11 (start at RuleEngine.java:431) failed in 0.094 s due to Job aborted due to stage failure: Task 0 in stage 11.0 failed 1 times, most recent failure: Lost task 0.0 in stage 11.0 (TID 8, localhost, executor driver): java.lang.NoClassDefFoundError: Could not initialize class kafka.utils.Log4jController$

Edit:

I was able to resolve the issue by changing the kafka client version in pom.xml

      <dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.11.0</version>
  </dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.11</artifactId>
  <version>2.3.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
    <version>0.8.2.2</version>
</dependency>

score 4 · Accepted Answer · answered Mar 10 '18 at 10:53

4

Examining your pom, the problem appears to be you are using kafka 1.0.0 but using spark-streaming-kafka-0-8, which is expecting kafka 0.8. Indeed, searching for kafka.utils.Log4jController reveals it was part of the kafka-clients library in versions 0.8.1 and 0.8.2 but not in later versions. I'm no expert on Spark but I think you just need to find a version of spark-streaming-kafka library that matches your kafka version. Hope that helps

answered Mar 10 '18 at 10:53

Michal Borowiecki

4,244
1
11
18

You are right. I changed the kafka version to 0.8.2.2 and it worked! Thank you! – Poojaa Karaande Mar 13 '18 at 08:12

score 0 · Answer 2 · answered Mar 10 '18 at 05:57

Your main problem is by throwing noClassDefFoundError:

How to resolve java.lang.NoClassDefFoundError:

Class is not available in Java Classpath.

You might be running your program using jar command and class was not defined in manifest file's ClassPath attribute.

Any start-up script is overriding Classpath environment variable.

Because NoClassDefFoundError is a sub class of java.lang.LinkageError it can also come if one of it dependency like native library may not available.

Check for java.lang.ExceptionInInitializerError in your log file. NoClassDefFoundError due to failure of static initialization is quite common.

If you are working in J2EE environment than visibility of Class among multiple Classloaders can also cause java.lang.NoClassDefFoundError, see examples and scenario section for detailed discussion.

which you can follow this link for more detail: noClassDefFoundError

Kafka with Spark throw Could not initialize class kafka.utils.Log4jController Error

2 Answers2