0

I don't have hadoop install in my system but I have winutils and I am getting this error while running hive in spark in my local:-

Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:869)
at com.sebi.count.ScalaSix$.main(ScalaSix.scala:14)
at com.sebi.count.ScalaSix.main(ScalaSix.scala)

My code is:-

package com.sebi.count

import org.apache.spark.sql.SparkSession

object ScalaSix {
  
 def main(args: Array[String]): Unit = {
   System.setProperty("hadoop.home.dir", "C:/winutils")
   val warehouseLocation = "spark-warehouse"
   val spark = SparkSession.builder().appName("Spark Hive Example")
     .config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate()
   spark.sparkContext.setLogLevel("ERROR")
   val df = spark.sql("CREATE TABLE IF NOT EXITS NAME(Id Int, Name String, Age Int,Salary Int)")
   val df1  = spark.sql("LOAD DATA LOCAL INPATH 'C:/Users/rittde/IdeaProjects/untitled/src/main/resources' " +
     "into table Name")
   val df2 = spark.sql("select * from Name")
   df2.show()
 }

}

My sbt dependency are:-

name := "untitled"
version := "0.1"
    
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.0" %"provided"

Can you please suggest how to resolve the error. Thanks in advance.

Vincent Doba
  • 4,343
  • 3
  • 22
  • 42
Rittik De
  • 1
  • 1
  • 1
  • But the ans that was posted didn't solve my doubt and the error is still their. – Rittik De Jan 25 '19 at 10:58
  • You can try to include Hive jars in class path before compilation, seems like from error message it is unable find required classes. Try to added hive related library dependencies and see if works. – Ajay Kharade Jan 25 '19 at 13:51

1 Answers1

2

I have the same error. I solved it by changing

libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.3" % "provided"

to

libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.3"

in my case. Here is a link explain sbt % "provide" configuration. SBT assembly jar exclusion

steadweb
  • 15,364
  • 3
  • 33
  • 47
Ma C
  • 41
  • 2