Unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16 when "sbt update" in scalding

Question

After getting code from git using clone https://github.com/twitter/scalding.git and doing ./sbt update I get:

::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found

And later:

sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:213)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)

I have removed ~/.sbt directory, installed latest Scala and cloned repository anew. The problem remains.

My configuration:

[dk@localhost scalding]$ scala -version
Scala code runner version 2.10.3 -- Copyright 2002-2013, LAMP/EPFL

[dk@localhost scalding]$ uname -a
Linux localhost.localdomain 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

CentOS 6.5 (Final)

What can be possibly wrong with ./sbt update? Should I install the dependency hadoop-lzo separately and set CLASSPATH to install Scalding? Any other solutions?

Here is the complete installation log:

[dk@localhost wks]$ cd scalding
[dk@localhost scalding]$ ./sbt update
Downloading sbt launcher for 0.13.0:
  From  http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.0/sbt-launch.jar
    To  /home/dk/.sbt/launchers/0.13.0/sbt-launch.jar
Getting org.scala-sbt sbt 0.13.0 ...
:: retrieving :: org.scala-sbt#boot-app
    confs: [default]
    43 artifacts copied, 0 already retrieved (12440kB/229ms)
Getting Scala 2.10.2 (for sbt)...
:: retrieving :: org.scala-sbt#boot-scala
    confs: [default]
    5 artifacts copied, 0 already retrieved (24390kB/150ms)
[info] Loading project definition from /home/dk/wks/scalding/project
[info] Updating {file:/home/dk/wks/scalding/project/}scalding-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 3 Scala sources to /home/dk/wks/scalding/project/target/scala-2.10/sbt-0.13/classes...
[warn] there were 10 deprecation warning(s); re-run with -deprecation for details
[warn] there were 2 feature warning(s); re-run with -feature for details
[warn] two warnings found
[info] Set current project to scalding (in build file:/home/dk/wks/scalding/)
[info] Updating {file:/home/dk/wks/scalding/}scalding-args...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-date...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}maple...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-core...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-json...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-parquet...
[info] Resolving com.hadoop.gplcompression#hadoop-lzo;0.4.16 ...
[warn]  module not found: com.hadoop.gplcompression#hadoop-lzo;0.4.16
[warn] ==== local: tried
[warn]   /home/dk/.ivy2/local/com.hadoop.gplcompression/hadoop-lzo/0.4.16/ivys/ivy.xml
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== snapshots: tried
[warn]   http://oss.sonatype.org/content/repositories/snapshots/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== releases: tried
[warn]   http://oss.sonatype.org/content/repositories/releases/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Concurrent Maven Repo: tried
[warn]   http://conjars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Clojars Repository: tried
[warn]   http://clojars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Twitter Maven: tried
[warn]   http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] Updating {file:/home/dk/wks/scalding/}scalding-jdbc...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-avro...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-commons...
[info] Resolving com.hadoop.gplcompression#hadoop-lzo;0.4.16 ...
[warn]  module not found: com.hadoop.gplcompression#hadoop-lzo;0.4.16
[warn] ==== local: tried
[warn]   /home/dk/.ivy2/local/com.hadoop.gplcompression/hadoop-lzo/0.4.16/ivys/ivy.xml
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== snapshots: tried
[warn]   http://oss.sonatype.org/content/repositories/snapshots/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== releases: tried
[warn]   http://oss.sonatype.org/content/repositories/releases/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Concurrent Maven Repo: tried
[warn]   http://conjars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Clojars Repository: tried
[warn]   http://clojars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Twitter Maven: tried
[warn]   http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] Updating {file:/home/dk/wks/scalding/}scalding-repl...
[info] Resolving org.codehaus.jackson#jackson-core-asl;1.8.8 ...
[info] Done updating.
sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:213)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
    at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
    at 
....
.... many lines of exception stack-trace are skipped ...
....

[error] (scalding-commons/*:update) sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[error] (scalding-parquet/*:update) sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[error] Total time: 53 s, completed Feb 14, 2014 4:51:51 PM
[dk@localhost scalding]$ ./sbt update^C
[dk@localhost scalding]$ uname -a
Linux localhost.localdomain 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[dk@localhost scalding]$ scala -version
Scala code runner version 2.10.3 -- Copyright

The problem is that you're using CentOS instead of the god blessed Debian Joking apart, that dependency comes from http://maven.twttr.com so make sure it is listed in repositories and is not blocked somehow by your host (e.g. can you open it in a browser?). Having said this, on my system `./sbt update` on a fresh repo runs flawlessly. — om-nom-nom, Feb 14 '14 at 14:11
@om-nom-nom The repo is already defined in `project/Build.scala` in the project https://github.com/twitter/scalding/blob/develop/project/Build.scala#L32 so there must be something different. — Jacek Laskowski, Feb 14 '14 at 14:56
@anton-ashanin you don't need to install Scala to run SBT projects. I'd go as far as to say you should remove it and `sbt console` is your buddy now. — Jacek Laskowski, Feb 14 '14 at 14:58
@anton-ashanin Can you paste the entire `sbt update` output? I'm pretty there's a bit more details on the root cause of the issue you face. — Jacek Laskowski, Feb 14 '14 at 14:59
@om-nom-nom You were right. This happens at my company environment and their it-security blocked access to maven.twttr.com. Debian is my favorite too, but I have to use CentOS as a project requirement. Notwithstanding with Debian at home I have some other installation problems with Scalding that I hope to write about later today. — Anton Ashanin, Feb 14 '14 at 15:18
@Jacek Laskowski Added installation log to my original question — Anton Ashanin, Feb 14 '14 at 15:29
Could be that you're blocked to access http://maven.twttr.com repository and hence http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom? Can you check using your browser and when it's worked `curl` from the command line? — Jacek Laskowski, Feb 15 '14 at 21:22

score 11 · Answer 1 · answered Sep 11 '14 at 10:27

11

SBT is unable to fetch com.hadoop.gplcompression#hadoop-lzo;0.4.16 from resolvers defined in your build.sbt

Adding maven.twttr.com: "Twitter Maven Repo" at "http://maven.twttr.com" should work.

answered Sep 11 '14 at 10:27

Hanginbat

111
1
4

This worked for me (it should be marked as the correct answer). – marios Mar 04 '16 at 23:09

score 9 · Answer 2 · answered Feb 01 '18 at 10:03

9

I don't think that would work anymore, use: "https://maven.twttr.com" instead. (Notice the https) They changed it in Jan 2018

Hope it helps

answered Feb 01 '18 at 10:03

Nir Hedvat

870
7
7

score 0 · Answer 3 · answered Nov 15 '19 at 14:03

Today I got a similar error but in lein. I come through a lot of articles and Jira issue. All recommend put https in URL and tag twitter repository directly. But this does not help. Does someone know another way?

list of checked sources.

https://stackoverflow.com/questions/21781028/unresolved-dependency-com-hadoop-gplcompressionhadoop-lzo0-4-16-when-sbt-upd
https://issues.apache.org/jira/browse/PARQUET-1556

Unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16 when "sbt update" in scalding

3 Answers3