0

I'm fairly new to scala / java / sbt and this is my first time debugging a discrepancy between code ran in 'scala' through sbt and the same code ran from a compiled jar. I have some Scala code which utilizes the AWS Java SDK to make some S3 requests. When I run my code in "Scala" using sbt or Intellij the code succeeds as intended. If I build a fat jar using sbt and run the same code in "Java" using the jar, the code fails. I debugged the failure and can attribute it to missing ExecutionInterceptors in the interceptor chain:

// Logging the code ran from 'scala'
2022-12-21 14:09:33,861 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -  Creating an interceptor chain that will apply interceptors in the following order: 
[software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@5965844d, 
software.amazon.awssdk.core.internal.interceptor.SyncHttpChecksumInTrailerInterceptor@6d4a65c6, 
software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@aa004a0, 
software.amazon.awssdk.core.internal.interceptor.AsyncRequestBodyHttpChecksumTrailerInterceptor@4c98a6d5, 
software.amazon.awssdk.core.internal.interceptor.HttpChecksumInHeaderInterceptor@392a04e7, 
software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@7f02251, 
software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@dffa30b, 
software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@4d8126f, 
software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@6d3c232f, 
software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@6b587673, 
software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@1bcf67e8, 
software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5f404594, 
software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@53692008, 
software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7b2a3ff8, 
software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@1bbae752, 
software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@460b6d54, 
software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@5cf87cfd, 
software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@76075d65, 
software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@3a4ba480, 
software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@27b71f50, 
software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@383790cf, 
software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@74971ed9, 
software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@131fcb6f, 
software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@ccd1bc3]

// Logging the same code ran from 'java'
2022-12-21 14:15:06,281 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -  Creating an interceptor chain that will apply interceptors in the following order: 
[software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@147e2ae7, 
software.amazon.awssdk.core.internal.interceptor.SyncHttpChecksumInTrailerInterceptor@448c8166, 
oftware.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@4470fbd6, 
software.amazon.awssdk.core.internal.interceptor.AsyncRequestBodyHttpChecksumTrailerInterceptor@15d49048, 
software.amazon.awssdk.core.internal.interceptor.HttpChecksumInHeaderInterceptor@7098b907, 
software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@503f91c3, 
software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@13526e59, 
software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@2235eaab, 
software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@17503f6b]

All of the interceptors of the type software.amazon.awssdk.services.s3.internal.handlers are missing from the code ran in the jar.

I'm not sure if this is directly related. My build.sbt has the following merge strategies:

  assembly / assemblyMergeStrategy := {
    // Don't discard Geotools META-INF service files
    case PathList("META-INF", xs @ _*) =>
      (xs map { _.toLowerCase }) match {
        case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
          MergeStrategy.discard
        case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
          MergeStrategy.discard
        case "plexus" :: xs =>
          MergeStrategy.discard
        case "services" :: xs =>
          MergeStrategy.filterDistinctLines
        case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
          MergeStrategy.filterDistinctLines
        case _ => MergeStrategy.discard
      }
    case PathList("reference.conf", xs @ _*) => MergeStrategy.concat
    case PathList("version.conf", xs @ _*)   => MergeStrategy.concat
    case x                                   => MergeStrategy.first
  },

When debugging compiling my jar with sbt assembly I noticed this line: [debug] Merging 'software/amazon/awssdk/services/s3/execution.interceptors' with strategy 'first' Is this related to why i'm missing the interceptors? or might there be some other cause for missing these handlers, such as a missing dependency?

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
jboges
  • 141
  • 1
  • 10
  • 1
    https://stackoverflow.com/questions/74800073/drools-fat-jar-nullpointer-kieservices https://stackoverflow.com/questions/74809158/run-drools-kie-project-from-fat-jar – Dmytro Mitin Dec 22 '22 at 07:27
  • 1
    https://stackoverflow.com/questions/74879217/sbt-assembly-and-lucene-an-spi-class-of-type-org-apache-lucene-codecs-codec-wit – Dmytro Mitin Dec 22 '22 at 08:15
  • 1
    Please provide the reproduction i.e. `build.sbt` with dependencies (not only assembly strategy part), what code throws the errors, how you're checking that something is missing etc. – Dmytro Mitin Dec 22 '22 at 08:27
  • https://stackoverflow.com/questions/25144484/sbt-assembly-deduplication-found-error – Dmytro Mitin Jan 14 '23 at 03:43

1 Answers1

1

I solved this question, the issue was that the aws s3 interceptors were getting merged out, I added this line to the merge strategy and it solved the problem:

case PathList(
        "software",
        "amazon",
        "awssdk",
        "services",
        "s3",
        "execution.interceptors",
        xs @ _*
        ) =>
      MergeStrategy.concat
jboges
  • 141
  • 1
  • 10