How to append a resource jar for spark-submit?

Question

My spark application depends on adam_2.11-0.20.0.jar, every time I have to package my application with adam_2.11-0.20.0.jar as a fat jar to submit to spark.

for example, my fat jar is myApp1-adam_2.11-0.20.0.jar,

It's ok to submit as following

spark-submit --class com.ano.adam.AnnoSp myApp1-adam_2.11-0.20.0.jar

It reported Exception in

thread "main" java.lang.NoClassDefFoundError:

org/bdgenomics/adam/rdd using --jars

spark-submit --class com.ano.adam.AnnoSp myApp1.jar --jars adam_2.11-0.20.0.jar

My question is how to submit using 2 separate jars without package them together

spark-submit --class com.ano.adam.AnnoSp myApp1.jar adam_2.11-0.20.0.jar

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

2

Add all jars in one folder and then do like below...

Option 1 :

I think Better way of doing this is

$SPARK_HOME/bin/spark-submit \
--driver-class-path  $(echo /usr/local/share/build/libs/*.jar | tr ' ' ',') \
--jars $(echo /usr/local/share/build/libs/*.jar | tr ' ' ',')

in this approach, you wont miss any jar by mistake in the classpath hence no warning should come.

Option 2 see my anwer:

spark-submit-jars-arguments-wants-comma-list-how-to-declare-a-directory

Option 3 : If you want to do programmatic submit by adding jars through API its possible.Here Im not going to details of it.

edited Jun 20 '20 at 09:12

Community

1
1

answered Nov 25 '16 at 03:13

Ram Ghadiyaram

28,239
13
95
121

was it helpful? your resource jar as well you can copy to that folder – Ram Ghadiyaram Nov 25 '16 at 06:28
spark-submit --driver-class-path adam_2.11-0.20.0.jar --class com.ano.adam.AnnoSp myApp1.jar works for spark2.0.1 – Bigo Nov 25 '16 at 06:40

How to append a resource jar for spark-submit?

1 Answers1

Option 1 :

Option 2 see my anwer:

Option 3 : If you want to do programmatic submit by adding jars through API its possible.Here Im not going to details of it.

Linked