2

I am trying to start a new SBT Scala project and have the following in build.sbt file:

name := "ScalaKafkaStreamsDemo"
version := "1.0"
scalaVersion := "2.12.1"

libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1" artifacts(Artifact("javax.ws.rs-api", "jar", "jar"))

libraryDependencies += "org.apache.kafka" %% "kafka" % "2.0.0"
libraryDependencies += "org.apache.kafka" % "kafka-streams" % "2.0.0"

So according to the GitHub repo, in 2.0.0 I should see the Scala classes/functions etc etc that I want to use, however they just don't seem to be available. Within IntelliJ I can open up the kafka-streams-2.0.0.jar, but I don't see any Scala classes.

Is there another JAR I need to include?

Just while we are on the subject of extra JARs, does anyone know what JAR I need to include to be able to use the EmbeddedKafkaCluster?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
sacha barber
  • 2,214
  • 1
  • 24
  • 37

3 Answers3

7

The artifact you need is kafka-streams-scala:

libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.1"

(please use 2.0.1, or even better 2.1.0, as 2.0.0 has some scala API bugs)

To answer your latter question, it's in the test-jar, which you can address using a classifier:

libraryDependencies += "org.apache.kafka" %% "kafka-streams" % "2.0.1" % "test" classifier "test"

But note that this is an internal class and subject to change (or removal) without notice. If at all possible, it's highly recommended that you use the TopologyTestDriver in the test-utils instead:

libraryDependencies += "org.apache.kafka" %% "kafka-streams-test-utils" % "2.0.1" % "test"
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
John
  • 726
  • 4
  • 5
  • Adding the line you described in the ticket, for `kafka-streams-scala 2.1.0` also works. But without it I also cannot resolve the javax.ws.rs dependency. – zaxme Dec 17 '18 at 15:59
  • Ah, thanks for the ticket @sacha, I was able to repro the problem, and it seems to be a misconfiguration of the `javax.ws.rs-api` pom. A quick hacky workaround is to fill in the variable they failed to put in their pom. For example: `sbt -Dpackaging.type=jar compile`. I left a comment on https://issues.apache.org/jira/browse/KAFKA-7741 explaining why. – John Dec 18 '18 at 00:29
3

It looks like you face the issue of unresolved javax.ws.rs-api dependency that happens with some Java projects that are direct or transitive dependencies of Scala projects that use sbt. I've faced it with Scala projects that use Apache Spark and recently with Kafka Streams (with and without the Scala API).

A workaround that has worked fine for me is to simply exclude the dependency and define it again explicitly.

excludeDependencies += ExclusionRule("javax.ws.rs", "javax.ws.rs-api")
libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1.1"

Make sure that you use the latest and greatest of sbt (i.e. 1.2.7 as of the time of this writing).

With that said, the dependencies in build.sbt should be as follows:

scalaVersion := "2.12.8"

val kafkaVer = "2.1.0"
libraryDependencies += "org.apache.kafka" % "kafka-streams" % kafkaVer
libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % kafkaVer

excludeDependencies += ExclusionRule("javax.ws.rs", "javax.ws.rs-api")
libraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1.1"

Within IntelliJ I can open up the kafka-streams-2.0.0.jar, but I don't see any Scala classes. Is there another JAR I need to include?

The following dependency is all you need:

libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % kafkaVer
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
0

You can also use following workaround, that works in my case - more details here

import sbt._
object PackagingTypePlugin extends AutoPlugin {
  override val buildSettings = {
    sys.props += "packaging.type" -> "jar"
    Nil
  }
}
Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30