1

Current when I run spark-submit I provide a whole bunch to paths of jars followed by the '--jars' option:

./spark\-submit --class "AppName" --master spark://server24:7077 --jars /path1.jar,path2.jar,path3.jar /pathAppName_2.10-1.0.jar arg1 arg2

Is there a cleaner way to include the jars files followed by --jar in the command above?

I tried adding them to spark.driver.extraClassPath in spark-defaults.conf but that does not seem to help. Couldn't find anything in the spark documentation otherwise.

Anyone know?

user3376961
  • 867
  • 2
  • 12
  • 17

2 Answers2

1

You can specify jars that you depend on when creating a SparkContext:

val conf = new SparkConf()
             .setMaster('local[*]')
             .setAppName('example')
             .setJars(Array('/path/to/dependencies/first.jar',
                            '/path/to/dependencies/second.jar'))

val sc = new SparkContext(conf)

This is basically what's happening under the covers when you use the --jars argument of spark-submit.

dayman
  • 680
  • 5
  • 10
0

The way I solved this in my Java Spark app was by using the maven shade plugin to create a fat-packed jar with all of the external dependencies in it. Otherwise, if you're using scala, this link may help you. For java, I would reference this.

In terms of another method of doing this out of the box with Spark, I don't think there's a cleaner way - at least if there is I never found one.

Community
  • 1
  • 1
Enigmatic Cipher
  • 98
  • 1
  • 7