0

My question is very similiar to the one found here: NoClassDefFoundError Apache Storm I have tried all the solutions posted there and I do not think the issue is the same.

I can compile my java code to a .jar without issue, but when I run it I get the below error. I included all the dependencies in the pom, yet it still can not find it. The code is almost directly copied from the Apache Storm documentation, to remove any other aspects that might give issues.

Any ideas would be greatly appreciated.

I get the following error message when trying to run my topology on my storm cluster:

Running: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/hadoop/storm -Dstorm.log.dir=/home/hadoop/stormLog -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp /home/hadoop/storm/*:/home/hadoop/storm/lib-worker/*:/home/hadoop/storm/extlib/*:twitterTest2.jar:/home/hadoop/storm/conf:/home/hadoop/storm/bin: -Dstorm.jar=twitterTest2.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} processingT.TTopology
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/storm/kafka/spout/KafkaSpout
    at processingT.TTopology.run(TTopology.java:24)
    at org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68)
    at processingT.TTopology.main(TTopology.java:16)
Caused by: java.lang.ClassNotFoundException: org.apache.storm.kafka.spout.KafkaSpout
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 3 more

I have this in my pom:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.storm.processing</groupId>
  <artifactId>processingT</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <dependencies>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-client</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>2.2.0</version>
    </dependency>   
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>2.6.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka-client</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-kafka</artifactId>
      <version>1.2.3</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-server</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.13</artifactId>
      <version>2.6.0</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.zookeeper</groupId>
          <artifactId>zookeeper</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
      </exclusions>
      </dependency>
  </dependencies>
  
  
</project>

And the java code is:

package processingT;

import org.apache.storm.topology.ConfigurableTopology;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.kafka.spout.KafkaSpout;
import org.apache.storm.kafka.spout.KafkaSpoutConfig;
import org.apache.storm.tuple.Fields;

public class TTopology extends ConfigurableTopology {
    public static void main(String[] args) throws Exception {
        ConfigurableTopology.start(new TTopology(), args);
    }

    @Override
    protected int run(String[] args) throws Exception {
        String port = "9092";
        
        TopologyBuilder tp = new TopologyBuilder();
        tp.setSpout("kafka_spout", new KafkaSpout<>(KafkaSpoutConfig.builder("ip:" + port, "topic").build()), 1);
        tp.setBolt("countT", new CountTBolt(), 12).fieldsGrouping("kafka_spout", new Fields("word"));

        conf.setDebug(true);

        String topologyName = "count";

        conf.setNumWorkers(3);

        if (args != null && args.length > 0) {
            topologyName = args[0];
        }
        return submit(topologyName,conf,tp);
    }
}
  • Which jar file contains class `ApacheStorm`? Is it on the classpath when you run the program? – Jim Garrison Dec 20 '20 at 19:16
  • Im running it on a deployed Apache-Storm cluster, I ran som other topologies on it to make sure it works, and they ran without issue, so that part should be ok. Edit: The cluster is deployed with the binaries from the Apache Storm download page/mirror. – BlueNebulae Dec 20 '20 at 19:30
  • KafkaSpout seems to be in its own jar file. Is `storm-kafka-n.n.n.jar` deployed in the cluster? https://findjar.com/class/org/apache/storm/kafka/KafkaSpout.html – Jim Garrison Dec 20 '20 at 20:19
  • It is, it is Included in the extlib-daemon folder – BlueNebulae Dec 21 '20 at 00:11
  • Well, either it's not really there, or there's some classloader funny business going on and your code's classloader can't see the class. – Jim Garrison Dec 21 '20 at 01:20
  • Hmm, ye, might just have to try and do a completely clean setup, see if I messed up something with the classloader or the like. Thx for the helps either way – BlueNebulae Dec 21 '20 at 12:03
  • Btw, you don't need `kafka_2.13`. That is the Kafka server libraries, not the client – OneCricketeer Dec 21 '20 at 17:41
  • Ah, thx. Went a bit overboard with adding jars – BlueNebulae Dec 21 '20 at 19:34

0 Answers0