7

Is there any configuration property we can set it to disable / enable Hive support through spark-shell explicitly in spark 1.6. I tried to get all the sqlContext configuration properties with,

sqlContext.getAllConfs.foreach(println)

But, I am not sure on which property can actually required to disable/enable hive support. or Is there any other way to do this?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Krishna Reddy
  • 1,069
  • 5
  • 12
  • 18

2 Answers2

17

Spark >= 2.0

Enable and disable of Hive context is possible with config spark.sql.catalogImplementation

Possible values for spark.sql.catalogImplementation is in-memory or hive

SPARK-16013 Add option to disable HiveContext in spark-shell/pyspark

Spark < 2.0

Such a Spark property is not available in Spark 1.6.

One way to work it around is to remove Hive-related jars that would in turn disable Hive support in Spark (as Spark has Hive support when required Hive classes are available).

Community
  • 1
  • 1
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
  • Thanks @Jacek Laskowski. The same i tried to recreate sqlContext object without HiveContext, as val sqlContext = new SQLContext(sc) which disables hive connection properties. Thanks. – Krishna Reddy Jul 21 '17 at 05:52
  • So then I guess if you haven't fiddled around with `config`s, then on `Spark 2.x`, `spark-shell` already has `Hive`-support enabled as mentioned [here](https://community.hortonworks.com/answers/144192/view.html) [or was that specifically for `HDP`?] – y2k-shubham Nov 14 '18 at 09:27
  • @y2k-shubham `spark-shell` enables Hive support by default so the trick is to use `spark.sql.catalogImplementation` on command line while launching it. – Jacek Laskowski Nov 14 '18 at 11:18
5

You can enable hive support just by creating spark session but only in spark >=2.0:

val spark = SparkSession
  .builder()
  .appName("Spark Hive Example")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()

And here you can read how to configure hive on spark by changing hive and spark properties in hive-site.xml, spark-defaults.conf: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started - it must work with spark 1.6.1

Yehor Krivokon
  • 837
  • 5
  • 17
  • Thanks @yehor Krivokon. Editing hive-site.xml or spark-defaults.conf should be required whenever i want to disable or enable hive in saprk 1.6.0, which is bit difficulty whenever i want to alter this hive enable.This is what i tried temporarily to serve my purpose, Recreate sqlContext without HiveContext like, val sqlContext = new SQLContext(sc). – Krishna Reddy Jul 20 '17 at 12:47
  • In this case I think it's really better to use spark 2.0+ where SparkSession is – Yehor Krivokon Jul 20 '17 at 13:24
  • @YehorKrivokon I am using spark-sql-2.3.1 version , how to get hiveContext ? – BdEngineer Nov 01 '18 at 09:57
  • @user3252097 HiveContext is deprecated since Spark 2.0.0 Use SparkSession.builder.enableHiveSupport instead. More info: https://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/sql/hive/HiveContext.html – Yehor Krivokon Nov 01 '18 at 12:00
  • @Yehorkrivokon , thank you i have different error now https://stackoverflow.com/questions/53100404/how-to-fix-exception-while-running-locally-spark-sql-program-on-windows10-by-ena – BdEngineer Nov 01 '18 at 12:14
  • @user3252097 I've answered on your new question, hope the answer will help you. – Yehor Krivokon Nov 01 '18 at 13:01
  • @YehorKrivokon sorry your answer is not useful as the error is coming on my windows machine – BdEngineer Nov 01 '18 at 16:11
  • @RussS sir can you please have a look at the task. I am still geting the same error..... – BdEngineer Nov 01 '18 at 18:35