0

I have an SBT project with Scala 2.12, with "za.co.absa" %% abris %% "4.2.0" as one of the dependencies. Transitively it brings kafka-clients dependency of version 5.3.4-css.

At the same time for tests I'm using embedded-kafka dependency: "io.github.embeddedkafka" %% "embedded-kafka" % "3.4.0" % Test. And tests are failing with errors (stack trace example):

java.lang.NoSuchMethodError: org.apache.kafka.common.utils.Utils.enumOptions(Ljava/lang/Class;)[Ljava/lang/String;
[info]   at kafka.server.KafkaConfig$.<init>(KafkaConfig.scala:1288)
[info]   at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala)
[info]   at io.github.embeddedkafka.ops.KafkaOps.startKafka(kafkaOps.scala:34)
[info]   at io.github.embeddedkafka.ops.KafkaOps.startKafka$(kafkaOps.scala:24)
[info]   at io.github.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:52)
[info]   at io.github.embeddedkafka.ops.RunningKafkaOps.startKafka(kafkaOps.scala:82)
[info]   at io.github.embeddedkafka.ops.RunningKafkaOps.startKafka$(kafkaOps.scala:75)
[info]   at io.github.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:52)
[info]   at io.github.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:70)
[info]   at com.careem.pi.optimus.jobs.BaseJobTest.beforeAll(BaseJobTest.scala:32)

Using dependencyTree I've figured out that kafka-clients is also a dependency coming from embedded-kafka with a different version, but it's overriden by dependency coming from abris:

┆ [info]   |   | +-org.apache.kafka:kafka-clients:3.4.0 (evicted by: 5.3.4-ccs)
┆ [info]   |   | +-org.apache.kafka:kafka-clients:5.3.4-ccs

Can I force SBT for using 3.4.0 version of kafka-clients for the Test scope, and 5.3.4-ccs version for the Compile scope?

Andrei Koch
  • 898
  • 1
  • 7
  • 23
  • 3
    https://stackoverflow.com/questions/74311373/can-you-import-a-separate-version-of-the-same-dependency-into-one-build-file-for – Dmytro Mitin Apr 04 '23 at 06:03
  • Haven't tried yet but it looks like a exactly the same problem – Andrei Koch Apr 04 '23 at 06:13
  • 1
    Yeah, I hope this should work. Although sometimes things connected with Spark can become trickier. So please for example try the solution with different subprojects and if this works for your use case please leave a feedback here and we'll close this one as a duplicate – Dmytro Mitin Apr 04 '23 at 06:18
  • 1
    Also https://stackoverflow.com/questions/75562485/nosuchmethoderror-scala-tools-nsc-settings-usejavacplscala-tools-nsc-settings https://stackoverflow.com/questions/75621915/reflection-to-call-method-that-had-its-name-changed-in-an-upgrade – Dmytro Mitin Apr 04 '23 at 07:24
  • 2
    Keep in mind that doing this reduce the reliability of your tests as they won't use the exact same code as production code. Another approach could be to use testcontainers instead of embedded Kafka. – Gaël J Apr 04 '23 at 17:30
  • @AndreiKoch Do you have any updates? – Dmytro Mitin Apr 06 '23 at 12:59

0 Answers0