0

I am running a job to extract data from Kafka into HBase using Spark/Scala. Jar is getting created without error but getting error when running spark-submit job, please help me.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/Put

  def load(gigi: DigiTbl, recordNum: Int, logEvery: Int): Put = {
    val rowKey = String.format("%s:%s", c1 c2)
    val put = new Put(Bytes.toBytes(rowKey))
    put.addColumn(columnFamily, Bytes.toBytes("c1"), Bytes.toBytes(digi.channel))
    if (recordNum % logEvery == 0) {
      LOG.info("Digital input: " + digi.toString)
      LOG.info("Put output: " + put.toJSON)
    }
    put
  }
AnumNuma
  • 31
  • 3
  • Have you passed on the hbase-client jar while submitting spark-submit? – Subash Kunjupillai May 10 '20 at 11:30
  • Yes, I did pass hbase-client.jar and I am getting another issue as well java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer, Google says this is because class is not linked/def found during run time. – AnumNuma May 11 '20 at 02:58
  • Did you check this https://stackoverflow.com/questions/54569033/spark-fails-with-noclassdeffounderror-for-org-apache-kafka-common-serialization ? – Subash Kunjupillai May 11 '20 at 08:23
  • 1
    Issue got resolved after adding --packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.3 into spark-submit command. Thanks for your help :). – AnumNuma May 12 '20 at 06:08

0 Answers0