4

I have a spark streaming Scala project which uses Apache NiFi receiver. The projects runs fine under Eclipse/Scala IDE and now I want to package it for deployment now. When I add it as

libraryDependencies += "org.apache.nifi" %% "nifi-spark-receiver" % "0.3.0"

sbt assumes it's a Scala library and tries to resolve it.

How doe I add NiFi receiver and all it's dependencies to project's SBT file?

Also, is it possible to pint dependencies to local directories instead of sbt trying to resolve?

Thanks in advance.

Here is my sbt file contents:

name := "NiFi Spark Test"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2" % "provided"
libraryDependencies += "org.apache.nifi" %% "nifi-spark-receiver" % "0.3.0"
elm
  • 20,117
  • 14
  • 67
  • 113
Igor K.
  • 915
  • 2
  • 12
  • 22
  • You can also try maven build [link](http://stackoverflow.com/questions/32265456/how-to-pre-package-external-libraries-when-using-spark-on-a-mesos-cluster/32403191#32403191) – Kaushal Nov 30 '15 at 06:41

2 Answers2

5

libraryDependencies += "org.apache.nifi" %% "nifi-spark-receiver" % "0.3.0"

Double % are used for adding scala version as suffix to the maven artefact. It is required because different scala compiler versions produces incompatible bytecode. If you are would like to use java library from maven, then you should use single % character

libraryDependencies += "org.apache.nifi" % "nifi-spark-receiver" % "0.3.0"
ayvango
  • 5,867
  • 3
  • 34
  • 73
  • Thank you. I did it and now getting the error bellow. Does it mean I need to add loj4j dependency? Also, is it possible to pint dependencies to local directories instead of sbt trying to resolve? [error] impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-log4j12;1.7.2 java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-log4j12;1.7.2 – Igor K. Nov 29 '15 at 20:19
1

I also found that I can put libraries the project depends on into the lib folder and they will be picked up during assembly.

Malte Schwerhoff
  • 12,684
  • 4
  • 41
  • 71
Igor K.
  • 915
  • 2
  • 12
  • 22