How to add external jar files to a spark scala project

Question

I am trying to use an LSH implementation of Scala(https://github.com/marufaytekin/lsh-spark) in my Spark project.I cloned the repository with some changes to the sbt file (added Organisation)

To use this implementation , I compiled it using sbt compile and moved the jar file to the "lib" folder of my project and updated the sbt configuration file of my project , which looks like this ,

Now when I try to compile my project using sbt compile , It fails to load the external jar file ,showing the error message "unresolved dependency: com.lendap.spark.lsh.LSH#lsh-scala_2.10;0.0.1-SNAPSHOT: not found". Am i following the right steps for adding an external jar file ? How do i solve the dependency issue

Please avoid posting screenshots and use the `code` formatting to put your sbt configuration. — Jonathan Taws, Jun 16 '16 at 10:11

score 1 · Answer 1 · edited May 23 '17 at 12:24

1

This issue isn't related to spark but to sbt configuration.

Make sure you followed the correct folder structure imposed by sbt and added your jar in the lib folder, as explained here - lib folder should be at the same level as build.sbt (cf. this post).

You might also want to check out this SO post.

edited May 23 '17 at 12:24

Community

1
1

answered Jun 16 '16 at 10:19

Jonathan Taws

1,168
11
24

I checked folder structure is fine . – tapas kumar Mahanta Jun 16 '16 at 10:21

Dazzler · Accepted Answer · 2016-06-16T16:41:34.160

1

As an alternative, you can build the lsh-spark project and add the jar in your spark application. To add the external jars, addJar option can be used while executing spark application. Refer Running spark application on yarn

edited Jun 16 '16 at 16:41

answered Jun 16 '16 at 13:09

Dazzler

807
9
11

How to add external jar files to a spark scala project

2 Answers2