2

I am running a Spark job locally. In my code, I am creating an AmazonS3 client. I am using a recent version of aws-java-sdk and spark. When I package my code with its dependencies and pass it to Spark submit utility, I get the following error.

Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.AmazonS3Client.builder()

I tried different online sources and could not get rid of the error. I even tried contacting the aws support, they asked me to add it to the spark-submit --packages. I tried that, still no luck.

CR241
  • 2,293
  • 1
  • 12
  • 30
Surabhi Dixit
  • 21
  • 3
  • Why don't you use a maven-shade-plugin to package (and shadow a jar if it conflicts with provided jar in the classpath) the aws-java-sdk jar and package it with your final jar? https://examples.javacodegeeks.com/enterprise-java/maven/maven-shade-plugin-example/ – user238607 Dec 06 '18 at 06:26
  • You can package all your dependencies as shown here : https://stackoverflow.com/questions/1729054/including-dependencies-in-a-jar-with-maven – user238607 Dec 06 '18 at 06:29
  • Thank you, that resolved my S3Client issue. – Surabhi Dixit Dec 06 '18 at 20:14

0 Answers0