0

I am running a topology to read from 100 kafka topics using 100 storm spouts and then 50 bolts emitting the data to my own data processing class, consisting of 10 instances/threads for each bolt. I am getting the following errors now and then(not always) only on some of the supervisor hosts(not always the same). I clearly have com.yammer.metrics.Metrics in my classpath.
However the error keeps bugging me:

java.lang.NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics
    at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:52)
    at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:25)
    at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:26)
    at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:37)
    at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:50)
    at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:50)
    at kafka.utils.Pool.getAndMaybePut(Pool.scala:61)
    at kafka.consumer.FetchRequestAndResponseStatsRegistry$.getFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:54)
    at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:39)
    at kafka.javaapi.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:34)
    at storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:43)
    at storm.kafka.PartitionManager.<init>(PartitionManager.java:58)
    at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
    at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
    at storm.kafka.KafkaSpoutWrapper.nextTuple(KafkaSpoutWrapper.java:74)
    at backtype.storm.daemon.executor$fn__8840$fn__8855$fn__8886.invoke(executor.clj:577)
    at backtype.storm.util$async_loop$fn__551.invoke(util.clj:491)
    at clojure.lang.AFn.run(AFn.java:22)
    at java.base/java.lang.Thread.run(Thread.java:833)

Any one having any ideas about this would be greatly appreciated! Thanks! The storm version I am using: 0.10.2 The kafka version I am using: kafka_2.10-0.8.2.1

The error is weird because it is not always happening and also not always happening for the same supervisor host, not for all the supervisor hosts. I dont know where to begin with

goodname
  • 1
  • "Clearly"? Based on what? – OneCricketeer Feb 15 '23 at 18:44
  • What Kafka server version are you actually using? 0.8 is almost 7+ years old! You never upgraded? – OneCricketeer Feb 15 '23 at 18:45
  • @OneCricketeer Yes, and I can not upgrade kafka for now for legacy reasons. – goodname Feb 15 '23 at 18:46
  • @OneCricketeer I did checked the class file exists in the topology jars. And not all the supervisor hosts are failling, some of them is working correctly, this is another proof – goodname Feb 15 '23 at 18:47
  • I dont use Strom, but I know in Hadoop/Spark, there is a config like `user.classpath.first`. Without this, it'll always use the JARs on the servers **before** any of those added in the "user JAR" – OneCricketeer Feb 15 '23 at 18:57
  • java.lang.NoClassDefFoundError This exception indicates that the JVM looked in its internal class definition data structure for the definition of a class and did not find it. This is different than saying that it could not be loaded from the classpath. Here you can expand information: https://stackoverflow.com/questions/34413/why-am-i-getting-a-noclassdeffounderror-in-java/5756989#5756989 – borchvm Feb 16 '23 at 10:49

0 Answers0