Unable to run spark sql through shell script

Question

I am unable to query a table in spark through shell script. But if i am running through command line, i am able to get the result. Problem arises when i insert those command in shell and trying to run.

Created a shell script :

vi test.sh

Inserted below spark shell command

spark-shell

val results =sqlContext.sql("SELECT * from table_name ")

results.show()

It is entering into spark shell but not running the below two command

val results =sqlContext.sql("SELECT * from table_name ")

results.show()

The table is available in the DB....while running my actual query i gaveDB.table_name — Abhinandan, Sep 06 '19 at 04:29

pawinder gupta · Answer 1 · 2019-09-06T05:09:16.310

I am assuming that you are able to query the data using hive. You need to do the configuration so that spark sql can work with HDFS and hive. Execute the following step for establishing a connection between spark and hive.

Create the file hive-site.xml at location $SPARK_HOME/conf/hive-site.xml. Make the following entry in this file (change the value to point to the metastore server of your hive installation):

<configuration>
  <property>
    <name>hive.metastore.uris</name>
    <!--Make sure that <value> points to the Hive Metastore URI in your cluster -->
    <value>thrift://sandbox.hortonworks.com:9083</value>
    <description>URI for client to contact metastore server</description>
  </property>
</configuration>

The following links give more detail on this:

How to connect to remote hive server from spark

https://acadgild.com/blog/how-to-access-hive-tables-to-spark-sql

score 0 · Answer 2 · answered Sep 06 '19 at 11:48

0

Instead of writing as a shell script, you can write as a scala file and run the scala file.

file.scala

val results =sqlContext.sql("SELECT * from table_name ")

results.show()

spark-shell -i file.scala

answered Sep 06 '19 at 11:48

Gowtham SB

332
1
3
16

score 0 · Accepted Answer · answered Sep 06 '19 at 12:57

You can use Except to get spark-shell working in bash script.

OR create a file with .scala and copy all your spark commands there.

val results =sqlContext.sql("SELECT * from table_name ")

results.show()

System.exit(0)

use ' spark-shell -i script_name.scala ' to run your script in bash or directly on linux terminal.

System.exit(0)----- to get out from spark-shell

Unable to run spark sql through shell script

3 Answers3