sbt cannot resolve dependencies

Question

I'm running sbt behind a proxy (with configuration in a file in sbt's conf), and it is to unable to resolve certain org.apache.hadoop#hadoop-mapreduce-client* dependencies:

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-app;2.3.0: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-core;2.3.0: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.3.0: not found
sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-app;2.3.0: not found
unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-core;2.3.0: not found
unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.3.0: not found
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:278)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:175)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:157)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:151)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:151)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:128)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
    at sbt.IvySbt$$anon$4.call(Ivy.scala:64)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
    at sbt.IvySbt.withIvy(Ivy.scala:123)
    at sbt.IvySbt.withIvy(Ivy.scala:120)
    at sbt.IvySbt$Module.withModule(Ivy.scala:151)
    at sbt.IvyActions$.updateEither(IvyActions.scala:157)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1318)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1315)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$85.apply(Defaults.scala:1345)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$85.apply(Defaults.scala:1343)
    at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1348)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1342)
    at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
    at sbt.Classpaths$.cachedUpdate(Defaults.scala:1360)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1300)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1275)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-app;2.3.0: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-core;2.3.0: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.3.0: not found
[error] Use 'last' for the full log.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0

This is my sbt.build file

name := "SparkSandbox"
version := "0.1"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.5.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.3.0"
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.5"

I have run sbt clean and sbt update, as recommended here. My repositories file looks as described here.

However, it's still stuck when I try to compile because it cannot resolve the dependencies. What am I missing?

score 0 · Answer 1 · edited Jan 18 '18 at 14:21

0

sbt respects the usual environment variables for http proxy settings:

export JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=yourserver -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username -Dhttp.proxyPassword=password"

(That's of course, assuming Unix (Linux/OSX etc). On windows you could just set the same environment variable (%JAVA_OPTS%) as usual in the Windows way.)

Credit: How to use sbt from behind proxy?

edited Jan 18 '18 at 14:21

Azzabi Haythem

2,318
7
26
32

answered Nov 12 '15 at 12:17

S V

570
8
21

The configuration in the conf file works as sbt has already downloaded all other dependencies, so I doubt that's the issue. Anyway, I have no idea what the problem was because it solved itself... – Ian Nov 12 '15 at 12:45

score 0 · Answer 2 · answered Jun 28 '16 at 03:49

0

I am able to resolve this issue. Please stop all the deamons and start again. Below are steps i followed in my quick start VM.

1.sudo service hadoop-master stop

2.sudo service hadoop-master start

3.hadoop dfsadmin -safemode leave

Now start the spark using spark-shell command.
Please run your spark program now.

answered Jun 28 '16 at 03:49

Girish Bejawada

1

1. The machine I am running sbt on does not have Hadoop installed. 2. The Spark shell has nothing to do with it. – Ian Jun 28 '16 at 05:45

score 0 · Answer 3 · answered Jun 28 '16 at 07:55

0

add version at end of the groupID in dependency "org.apache.hadoop" % "hadoop-client_2.11" % "2.3.0"

answered Jun 28 '16 at 07:55

Sandeep Purohit

3,652
18
22

Not according to [Maven central](http://search.maven.org/#artifactdetails%7Corg.apache.hadoop%7Chadoop-client%7C2.3.0%7Cjar). – Ian Jun 28 '16 at 08:43
i laso facing same problem of unresolved dependency in sbt soo we can use resolver for that or just add full dependency with scala version – Sandeep Purohit Jun 28 '16 at 10:29

sbt cannot resolve dependencies

3 Answers3