spark shell (spark 3.0.0) adding packages confluent kafka 5.5.1 issues with javax.ws.rs-api

Question

My local win10 WSL back unbuntu, In ubuntu, i have installed spark3.0.0, confluent platform 5.5.1 (manually downloaded)

when I try to run either spark-shell or spark-submit (below is shell example)

spark-shell\
 --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,\
org.apache.spark:spark-avro_2.12:3.0.0,\
io.confluent:kafka-schema-registry:5.5.1,\
io.confluent:kafka-avro-serializer:5.5.1

they both have issues like below:

:::: WARNINGS
        [NOT FOUND  ] javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type} (15ms)

    ==== local-m2-cache: tried

      file:/home/min/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::              FAILED DOWNLOADS            ::

        :: ^ see resolution messages for details  ^ ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}

        ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: javax.ws.rs#javax.ws.rs-api;2.1.1!javax.ws.rs-api.${packaging.type}]

tried to add below, did not help.

 --repositories https://packages.confluent.io/maven/\

note: if i remove the last two confluent platform packages, it has no problems.

please help, maybe i have to downgrade either of the version.

no. no proxy. the end goal is to try to read avro message off confluent kafka, trying to follow this: https://stackoverflow.com/questions/48882723/integrating-spark-structured-streaming-with-the-confluent-schema-registry — soMuchToLearnAndShare, Sep 11 '20 at 12:05
I will give up this one for now. as It was a rabbit hole for my end goal. — soMuchToLearnAndShare, Sep 11 '20 at 15:05

score 1 · Answer 1 · answered Nov 09 '20 at 08:04

This is issue in sbt https://github.com/sbt/sbt/issues/3618

A workaround for now is to replace javax.ws.rs-api with jakarta.ws.rs-api.

Luckily for you confluent already done this in their libraries so if you use version 5.5.2 like this :

io.confluent:kafka-schema-registry:5.5.2,
io.confluent:kafka-avro-serializer:5.5.2

It should be fixed.

spark shell (spark 3.0.0) adding packages confluent kafka 5.5.1 issues with javax.ws.rs-api

1 Answers1

Linked