use sparkling-water via spark packages: com.google.guava... not found

Question

I'm trying to use H2O.ai's sparkling-water via spark packages. I'm following their guide: https://github.com/h2oai/sparkling-water#use-sparkling-water-via-spark-packages I'm on Hortonworks HDP 2.4 with Scala 2.10 and Spark 1.6.1. I put the following in my terminal:

spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.8,ai.h2o:sparkling-water-examples_2.10:1.6.8

However, it complains two artifacts not found. Here're the details:

:: problems summary ::
:::: WARNINGS
        [NOT FOUND  ] com.google.guava#guava;16.0.1!guava.jar(bundle) (1ms)
==== local-m2-cache: tried

  file:/root/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar

    [NOT FOUND  ] com.google.code.findbugs#jsr305;3.0.0!jsr305.jar (1ms)

==== local-m2-cache: tried

  file:/root/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar

    ::::::::::::::::::::::::::::::::::::::::::::::

    ::              FAILED DOWNLOADS            ::

    :: ^ see resolution messages for details  ^ ::

    ::::::::::::::::::::::::::::::::::::::::::::::

    :: com.google.guava#guava;16.0.1!guava.jar(bundle)

    :: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar

    ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: com.google.guava#guava;16.0.1!guava.jar(bundle), download failed: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1068)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Anyone has any experience how I can fix it? Thanks a lot.

score 1 · Answer 1 · answered Mar 01 '17 at 16:55

1

Removing ~/.m2 and ~/ivy2 fixed the issue with using ai.h2o:sparkling-water-core_2.10:1.6.8

answered Mar 01 '17 at 16:55

BlueFeet

2,407
4
21
24

I guess you mean `~/.ivy2` as it is a hidden directory. – kar09 May 30 '21 at 10:33

score 0 · Answer 2 · answered Feb 22 '17 at 01:49

If you look for the error "download failed: com.google.guava#guava;16.0.1!guava.jar" around, you will see it is mostly an environment specific issues where packages could not be downloaded during the application launched.

Your solution could be to download/install both the failed downloaded package separately something as below to get you going:

$ mvn install com.google.guava:guava:21.0:get -DrepoUrl=https://mvnrepository.com

score 0 · Answer 3 · answered Feb 27 '17 at 21:15

It looks like lowering the version of the jar files

spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.1,ai.h2o:sparkling-water-examples_2.10:1.6.1

or using the fatjar suggested in Section 5.1.1 of Sparkling Water Booklet

spark-shell --jars [your path]/sparkling-water-1.6.8/assembly/build/libs/sparkling-water-assembly-1.6.8-all.jar

solves the issue. Where sparkling-water-1.6.8 is downloaded from H2O

use sparkling-water via spark packages: com.google.guava... not found

3 Answers3