Run spark-shell command in shell script

Question

#!/bin/sh
spark-shell
import org.apache.spark.sql.SparkSession
val url="jdbc:mysql://localhost:3306/slow_and_tedious"
val prop = new java.util.Properties
prop.setProperty("user",”scalauser”)
prop.setProperty("password","scalauser123")
val people = spark.read.jdbc(url,"sat",prop)

The above commands are used to make a connection between Mysql and Spark using JDBC. But instead of writing these commands everytime I thought of make a script but when I run the above script it throws this error.

Answer 1

Create scala file named as test.scala with your code like below

import org.apache.spark.sql.SparkSession
val url="jdbc:mysql://localhost:3306/slow_and_tedious"
val prop = new java.util.Properties
prop.setProperty("user",”scalauser”)
prop.setProperty("password","scalauser123")
val people = spark.read.jdbc(url,"sat",prop)

Login to spark-shell using following command.

spark-shell --jars mysql-connector.jar

you can use following command to execute the code which you created above.

scala> :load /path/test.scala

shell script every time it launch sparkContext which takes more time to execute.

If you use above command it will just execute the code which is there in test.scala.

Since sparkContext will be loaded when you logging into spark-shell, time can be saved when you execute script.

Answer 2

Try this,

Write your code in a filex.txt

In your unix shell script include the following

cat filex.txt | spark-shell

Seemingly, you cant push the script in Background (using &)

Answer 3

you can paste your script in a file,then execute

spark-shell < {your file name}