Unresolved dependencies errors when trying to build self-contained application with Scala + Spark

Question

I am trying to follow the Self-Contained Applications Tutorial and trying to build using sbt and scala, but I run into the following errors when calling sbt package from C:\Users\00926133\test:

[info] Set current project to Simple Project (in build file:/C:/Users/00926133/test/)
[info] Updating {file:/C:/Users/00926133/test/}test...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[error] Unable to find credentials for [ @ XXXX].
[info] Resolving org.apache.spark#spark-core_2.10;1.4.0 ...
[info] Resolving org.apache.spark#spark-core_2.10;1.4.0 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.scala-lang#scala-library;2.10.4: java.util.NoSuchElementException
[warn]  :: org.apache.spark#spark-core_2.10;1.4.0: java.util.NoSuchElementException
[warn]  :: org.scala-lang#scala-compiler;2.10.4: java.util.NoSuchElementException
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.scala-lang:scala-library:2.10.4 ((sbt.Classpaths) Defaults.scala#L1203)
[warn]        +- simple-project:simple-project_2.10:1.0
[warn]      org.apache.spark:spark-core_2.10:1.4.0 (C:\Users\00926133\test\simple.sbt#L7-8)
[warn]        +- simple-project:simple-project_2.10:1.0
[warn]      org.scala-lang:scala-compiler:2.10.4
[warn]        +- simple-project:simple-project_2.10:1.0
sbt.ResolveException: unresolved dependency: org.scala-lang#scala-library;2.10.4: java.util.NoSuchElementException
unresolved dependency: org.apache.spark#spark-core_2.10;1.4.0: java.util.NoSuchElementException
unresolved dependency: org.scala-lang#scala-compiler;2.10.4: java.util.NoSuchElementException
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:291)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:188)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:165)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:132)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
    at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
    at sbt.IvySbt.withIvy(Ivy.scala:127)
    at sbt.IvySbt.withIvy(Ivy.scala:124)
    at sbt.IvySbt$Module.withModule(Ivy.scala:155)
    at sbt.IvyActions$.updateEither(IvyActions.scala:165)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1369)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1365)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$87.apply(Defaults.scala:1399)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$87.apply(Defaults.scala:1397)
    at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1402)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1396)
    at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
    at sbt.Classpaths$.cachedUpdate(Defaults.scala:1419)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1348)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1310)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[error] (*:update) sbt.ResolveException: unresolved dependency: org.scala-lang#scala-library;2.10.4: java.util.NoSuchElementException
[error] unresolved dependency: org.apache.spark#spark-core_2.10;1.4.0: java.util.NoSuchElementException
[error] unresolved dependency: org.scala-lang#scala-compiler;2.10.4: java.util.NoSuchElementException
[error] Total time: 1 s, completed Jun 26, 2015 9:13:40 AM

I have been successful in running spark-shell and scala, as well as the spark examples for python and R. I have also tried removing the .ivy folder.

My simple.sbt file:

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.4.0"

Other Information

scala -version == 2.10.4
sbt sbt-version == 0.13.8
My simple.sbt file is located at C:\Users\00926133\test
SimpleApp.scala is located at C:\Users\00926133\test\src\main\scala
Spark is residing at C:\Users\00926133\spark-1.4.0-bin-hadoop2.6

I have looked through several Stack Overflow posts, but none of the solutions shown there seem to get me any traction.

UPDATE
I was able to confirm this is a proxy related issue. However, I am still unable to build behind my proxy. This is on Windows and I have the following:

Environment Variables

SBT_OPTS = -Dhttp.proxyHost=<myhost> -Dhttp.proxyPort=<myport> -Dhttp.proxyUser=<myuser> -Dhttp.proxyPassword=<mypassword> -Dhttps.proxyHost=<myhost> -Dhttps.proxyPort=<myport> -Dhttps.proxyUser=<myuser> -Dhttps.proxyPassword=<mypassword> 
JAVA_OPTS = -Dhttp.proxyHost=<myhost> -Dhttp.proxyPort=<myport> -Dhttp.proxyUser=<myuser> -Dhttp.proxyPassword=<mypassword> -Dhttps.proxyHost=<myhost> -Dhttps.proxyPort=<myport> -Dhttps.proxyUser=<myuser> -Dhttps.proxyPassword=<mypassword>

I also add the following lines to my sbtconfig.txt located under ..\sbt\conf:

-Dhttp.proxyHost=<myhost>
-Dhttp.proxyPort=<myport>
-Dhttp.proxyUser=<myuser>
-Dhttp.proxyPassword=<mypassword>
-Dhttps.proxyHost=<myhost>
-Dhttps.proxyPort=<myport>
-Dhttps.proxyUser=<myuser>
-Dhttps.proxyPassword=<mypassword>

Bonus points for anyone who can (a) explain in layman's terms what sbt package is doing, (b) how to interpret the simple.sbt file, and/or (c) figure out how to resolve [error] Unable to find credentials for [ @ XXXX ] -- XXXX is my work's proxy-server.

score 3 · Accepted Answer · edited May 23 '17 at 12:23

I found the source code to your first error (chronologically) in the SBT source code accompanied with the comment "We're guaranteed to only get here if Ivy's authentication fails". Since the remaining errors are an inability to find dependencies, it looks like the proxy is causing the problem.

Did you see the question How to use sbt from behind proxy? Did you try those steps?

Ivy is used by SBT to cache dependencies locally, so that once you've fetched a particular version of a particular dependency you won't have to go to the Internet to fetch it again. In Windows, you can see the contents of your Ivy cache at C:\Users\<User Name>\.ivy2

It looks like SBT's inability to get through your company's proxy is causing it to be unable to fetch third party dependencies, thus causing the build to fail. If you haven't already tried that, try setting the proxy configuration on the SBT command line or in environment variables. Since you're using Windows, just add -Dhttp.proxyHost=yourserver -Dhttp.proxyPort=port ... to the SBT command line. Make sure you add all of the required parameters, depending on what the proxy requires. (Does it require a username/password?)

SBT is Scala's "Simple Build Tool." SBT files are written in Scala. The classes in SBT have operators defined that cause the build actions you request based on your build file. When you run sbt package you're asking the build tool to build and package the current project.

The Spark quick start explains the simple.sbt file at a high level. Generally, the SBT build file gives a project name and version, defines the required Scala version, lists third party dependencies, and generally speaking defines everything needed for the entire build and packaging process.

+1 Thank you. This is **definitely** a proxy issue. Despite many effort in trying to set `JAVA_OPTS` and `SBT_OPTS` and modifying the `sbtconfig.txt` file, I was unable to get things to build behind the proxy. However, I was able to build *off the proxy* and then successfully run the example. Really wish I could figure out the secret combo of options to get things building behind the proxy with sbt. Proxy settings have been a pain with other applications, but I've been able to resolve them. — JasonAizkalns, Jun 26 '15 at 18:28
Are you getting the same error as before "Unable to find credentials for [ @ XXXX]." or something different? — Eddie, Jun 26 '15 at 19:30

Unresolved dependencies errors when trying to build self-contained application with Scala + Spark

1 Answers1