1

I'm trying to build a Cascading job that uses Hive to perform some of the transformations. The cascading-hive project seems like a great starting point. I'm running Gradle 2.6 and Java 1.8. Per the documentation, I ran the following:

git clone https://github.com/Cascading/cascading-hive
cd cascading-hive
gradle install

Unfortunately, this returned an error. A jar was missing from my local maven repo:

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':testCompile'.
> Could not find hadoop-common-tests.jar (org.apache.hadoop:hadoop-common:2.6.0).
  Searched in the following locations:
      file:/Users/awoolford/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0-tests.jar

I searched for this jar on Maven Central and it was not there.

Can you see what needs to happen in order to build cascading-hive?

Update

I had a quick look in the jars provided by Cloudera, found their version of this jar (e.g. /opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/hadoop-common-2.6.0-cdh5.4.4-tests.jar), and copied it to /Users/awoolford/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0-tests.jar. gradle install now executes successfully, though this doesn't seem like a good general solution.

Alex Woolford
  • 4,433
  • 11
  • 47
  • 80
  • Looks like a typo, the artifact is [`hadoop-common-test`](http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common-test) – Tunaki Aug 17 '15 at 14:43
  • While using cloudera, I would suggest using their versions of jars. I recently faced issues with CDH 5.5.1 version jars not working correctly with Apache jars. I changed my pom / gradle script to pull the cloudera version of hadoop and hive (hadoop: 2.6.0-cdh5.5.1 and hive: 1.1.0-cdh5.5.1) from cloudera repo. – Prabodh Mhalgi Mar 11 '16 at 03:10

1 Answers1

0

Hadoop does not seem to be correctly downloaded in the local repo. I cannot pinpoint exactly what the exact problem in your case is. See How can I force gradle to redownload dependencies? for suggestions on how to purge.

PS. The jar is available in the maven central repository. See http://search.maven.org/#artifactdetails%7Corg.apache.hadoop%7Chadoop-common%7C2.6.0%7Cjar

Community
  • 1
  • 1
Gunnar Tångring
  • 395
  • 2
  • 7