Initialising HiveContext in Spark CLI

Question

When Initialising Spark in Command-line interface by default SparkContext is initialised as sc and SQLContext as sqlContext.

But I need HiveContext as I am using a function collect_list which is not supported by SparkContext, but is supported by HiveContext. Since HiveContext is a superclass of SparkContext ,it should have worked,but it isn't.

HOW DO I INITIALISE HiveContext in Scala using Spark CLI?

score 3 · Answer 1 · edited Mar 13 '18 at 09:40

3

You can do so by following the below steps:

import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)
val depts = sqlContext.sql("select * from departments")`

edited Mar 13 '18 at 09:40

t j

7,026
12
46
66

answered Mar 13 '18 at 09:19

Amit khandelwal

504
5
6

score 2 · Accepted Answer · edited May 23 '17 at 12:06

In spark-shell, sqlContext is an instance of HiveContext by default. You can read about that in my previous answer here.

Nevertheless, collect_list isn't available in spark 1.5.2. It was introduced in spark 1.6 so it's normal that you can find it.

Reference : https://github.com/apache/spark/blob/v1.6.2/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L213

Also you don't need to import org.apache.spark.sql.functions._ in the shell. It's imported by default.

Rockie Yang · Answer 3 · 2016-07-04T07:20:19.173

1

The sqlContext is a HiveContext

scala> sqlContext
res11: org.apache.spark.sql.SQLContext = org.apache.spark.sql.hive.HiveContext@4756c8f3

[Edit]

Import the functions before use it.

import org.apache.spark.sql.functions._

edited Jul 04 '16 at 07:20

answered Jul 04 '16 at 06:52

Rockie Yang

4,725
31
34

`error: not found: value collect_list` this is what the console says after I use **collect_list** function – indian_authority Jul 04 '16 at 07:10
do you imported correctly? import org.apache.spark.functions._ – Rockie Yang Jul 04 '16 at 07:12
`scala> import org.apache.spark.functions._ :31: error: object functions is not a member of package org.apache.spark import org.apache.spark.functions._ ^ ` **HOW DO I RECTIFY THIS** ? Please help – indian_authority Jul 04 '16 at 07:14
@indian_authority which version of spark do you have ? – eliasah Jul 04 '16 at 07:14
@elisah ` ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 /_/ ` – indian_authority Jul 04 '16 at 07:17
sorry my mistake. it should be import org.apache.spark.sql.functions._ – Rockie Yang Jul 04 '16 at 07:18

Initialising HiveContext in Spark CLI

3 Answers3