2

I've an SBT project that depends on

"com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop3-2.2.2"

which is bringing a recent version of google-api-services-storage.

I've also another dependency to Sparkling Water which is a fat jar that seems to bring a different version of google-api-services-storage packaged with the rest of this library classes inside that fat jar.

Now whenever I try to use gcs-connector it fails with this error:

Caused by: java.lang.NoSuchMethodError: com.google.api.services.storage.Storage$Objects$List.setIncludeTrailingDelimiter(Ljava/lang/Boolean;)Lcom/google/api/services/storage/Storage$Objects$List;
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.createListRequest(GoogleCloudStorageImpl.java:1401)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.listStorageObjectsAndPrefixes(GoogleCloudStorageImpl.java:1272)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.listObjectInfo(GoogleCloudStorageImpl.java:1443)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.lambda$getFileInfoInternal$8(GoogleCloudStorageFileSystem.java:1059)

It seems the class loader is picking up the classes from inside that fat jar.

In this scenario how can I shade the classes inside the fat jar? I tried the following but without success (see https://github.com/h2oai/sparkling-water/issues/2643)

settings = Seq(
  ...,
  libraryDependencies ++= Seq(...),
  assemblyShadeRules in assembly ++= Seq(
    ShadeRule.rename("com.google.api.services.storage.**" -> "shade.@1")
      .inLibrary("the fat jar group" % "the fat jar linbary" % "the fat jar version")
  )
)

Update: after shading google dependency in the original I did run into another dependency issue:

Caused by: java.lang.NoSuchMethodError: io.opencensus.trace.Span.addMessageEvent(Lio/opencensus/trace/MessageEvent;)V
    at com.google.api.client.http.OpenCensusUtils.recordMessageEvent(OpenCensusUtils.java:222)
    at com.google.api.client.http.OpenCensusUtils.recordSentMessageEvent(OpenCensusUtils.java:190)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1010)
    at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:304)
    at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:324)
    at com.google.cloud.hadoop.util.CredentialFactory$GoogleCredentialWithRetry.executeRefreshToken(CredentialFactory.java:170)
    at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:470)
    at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:201)
bachr
  • 5,780
  • 12
  • 57
  • 92
  • 1
    I don't think it is easy, usually you can only shade your own jar to avoid conflict with a transitive dependency, see https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc – Dagang Sep 17 '21 at 17:44
  • yeah that's what i released after trying hard to shade or even exclude the `google-api-services-storage` that's inside the fat jar. – bachr Sep 17 '21 at 18:19
  • Now i'm trying to avoid depending on the fat jar but instead picking up individual packages but even that is not straightforward as they don't publish one of the dependencies see https://github.com/h2oai/sparkling-water/issues/2643 – bachr Sep 17 '21 at 18:21
  • 1
    Maybe you can try unpacking the fat jar, removing the unwanted classes, then recreating the jar? https://stackoverflow.com/questions/16806112/how-to-remove-class-file-in-jar-file-and-change-it-with-my-own-implementation – Dagang Sep 17 '21 at 18:24
  • yeah that;s one way of doing it – bachr Sep 17 '21 at 18:51
  • Does Sparkling Water provide a non-fat jar? If it does, you can use it and exclude the conflicting transitive dependency. https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html#dependency-exclusions – Dagang Sep 18 '21 at 02:27
  • This might be useful https://stackoverflow.com/questions/38617107/exclude-class-or-package-inside-classpath-when-starting-an-application – Dagang Sep 19 '21 at 06:38
  • @Dagang they do publish some of the packages but not all of them that's why I had to use the fat jar. I think for the time being I will try to manually update the jar and remove the files – bachr Sep 20 '21 at 17:19
  • I solved the issue by shading google packages in the fat jar https://github.com/h2oai/sparkling-water/pull/2645 – bachr Sep 20 '21 at 20:50
  • Cool, good to know. – Dagang Sep 20 '21 at 21:21
  • @Dagang actually I run into another issue while using the shading approach, it seems I would need to basically shade every other library that google-api-services-storage depends on like in this case io.opencensus, see update above – bachr Sep 21 '21 at 00:47
  • I think best approach is not to depend on the fat jar at all, will try to figure out if I can get the private package published – bachr Sep 21 '21 at 00:51

0 Answers0