Apache sqoop is not compiling

Question

I am trying to compile apache sqoop from source code as I need to build an extension of it. The code compilation is failing.

Code version from Git - 1.6. - https://github.com/apache/sqoop
ivy dependency can't be resolved for  org.mortbay.jetty#jetty;6.1.26!jetty.zip

ant package command is giving following error

[ivy:resolve]       [FAILED     ] org.mortbay.jetty#jetty;6.1.26!jetty.zip:  (0ms)

You should not be compiling Sqoop yourself unless you are attempting to help develop it. Try downloading a binary distribution from http://www.apache.org/dyn/closer.cgi/sqoop/1.4.5. — generalpiston, Nov 23 '14 at 08:51
@abeaamase - Compiling your own open-source software really requires no justification. It's the whole point of being open source! Perhaps you're trying to add logging, or find and fix a bug. Perhaps some internal policy requires all production software to be built on-site after auditing the source. Perhaps you _want_ to help develop it, and compiling the original source is a perfectly reasonable first step - how else can you test a bugfix or feature you want to send upstream if you can't compile it? — Hercynium, Jul 22 '15 at 18:53
OP: I am looking for an answer to this issue myself. I suspect there is an upstream dependency causing this issue, and the solution may lie with the Ivy part of the build. I'll update when I know more. — Hercynium, Jul 22 '15 at 18:56
@Hercynium good point. I chose the wrong words here. I was trying to convey that extending Sqoop1 isn't the best idea (Sqoop2 would be better for that). Any ways, my follow through below should help. — generalpiston, Jul 22 '15 at 18:58

score 1 · Answer 1 · edited May 23 '17 at 10:24

1

A few things I'd try:

Make sure you have the right versions of ant and maven. From the COMPILING notes, Apache ant (1.7.1) at minimum is required.
Clear your ivy and maven cache.
Hit up the Sqoop mailing lists for further assistance.

edited May 23 '17 at 10:24

Community

1
1

answered Nov 20 '14 at 18:34

generalpiston

911
7
11

Hercynium · Answer 2 · 2015-07-24T20:44:55.023

I found this in my ~/.ivy2/com.cloudera.sqoop-sqoop-hadoop200.xml:

<module organisation="org.mortbay.jetty" name="jetty">
  <revision name="6.1.26" status="release" pubdate="20150715114025" resolver="fs" artresolver="default" homepage="" downloaded="false" searched="false" default="false" conf="compile, master(*), runtime, compile(*), runtime(*), master" position="183">
    <metadata-artifact status="no" details="" size="2189" time="0" location="/home/sscaffidi/.ivy2/org.mortbay.jetty/jetty/ivy-6.1.26.xml" searched="false" origin-is-local="true" origin-location="/home/sscaffidi/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.pom"/>
    <caller organisation="org.apache.hadoop" name="hadoop-common" conf="default, compile, runtime, master" rev="6.1.26.cloudera.2" rev-constraint-default="6.1.26.cloudera.2" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <caller organisation="org.apache.hbase" name="hbase-server" conf="default, compile, runtime, master" rev="6.1.26.cloudera.4" rev-constraint-default="6.1.26.cloudera.4" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="1.0.0-cdh5.4.4"/>
    <caller organisation="org.apache.hadoop" name="hadoop-hdfs" conf="default, compile, runtime, master" rev="6.1.26" rev-constraint-default="6.1.26" rev-constraint-dynamic="6.1.26" callerrev="2.2.0"/>
    <caller organisation="org.apache.hadoop" name="hadoop-yarn-server-web-proxy" conf="compile, runtime" rev="6.1.26.cloudera.2" rev-constraint-default="6.1.26.cloudera.2" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <caller organisation="org.mortbay.jetty" name="jetty-sslengine" conf="compile, runtime" rev="6.1.26.cloudera.4" rev-constraint-default="6.1.26.cloudera.4" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="6.1.26.cloudera.4"/>
    <caller organisation="org.apache.hadoop" name="hadoop-hdfs" conf="default, compile, runtime, master" rev="6.1.26.cloudera.4" rev-constraint-default="6.1.26.cloudera.4" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <artifacts>
            <artifact name="jetty" type="zip" ext="zip" status="failed" details="" size="0" time="0"/>
    </artifacts>
  </revision>
  <revision name="6.1.26.cloudera.4" status="release" pubdate="20150720102423" resolver="fs" artresolver="default" evicted="latest-revision" evicted-reason="" homepage="" downloaded="false" searched="false" default="false" conf="compile, master(*), runtime, compile(*), runtime(*), master" position="-1">
    <metadata-artifact status="no" details="" size="7312" time="0" location="/home/sscaffidi/.ivy2/org.mortbay.jetty/jetty/ivy-6.1.26.cloudera.4.xml" searched="false" origin-is-local="true" origin-location="/home/sscaffidi/.m2/repository/org/mortbay/jetty/jetty/6.1.26.cloudera.4/jetty-6.1.26.cloudera.4.pom"/>
    <evicted-by rev="6.1.26"/>
    <caller organisation="org.apache.hadoop" name="hadoop-common" conf="default, compile, runtime, master" rev="6.1.26.cloudera.2" rev-constraint-default="6.1.26.cloudera.2" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <caller organisation="org.apache.hbase" name="hbase-server" conf="default, compile, runtime, master" rev="6.1.26.cloudera.4" rev-constraint-default="6.1.26.cloudera.4" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="1.0.0-cdh5.4.4"/>
    <caller organisation="org.apache.hadoop" name="hadoop-yarn-server-web-proxy" conf="compile, runtime" rev="6.1.26.cloudera.2" rev-constraint-default="6.1.26.cloudera.2" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <caller organisation="org.mortbay.jetty" name="jetty-sslengine" conf="compile, runtime" rev="6.1.26.cloudera.4" rev-constraint-default="6.1.26.cloudera.4" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="6.1.26.cloudera.4"/>
    <caller organisation="org.apache.hadoop" name="hadoop-hdfs" conf="default, compile, runtime, master" rev="6.1.26.cloudera.4" rev-constraint-default="6.1.26.cloudera.4" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <artifacts>
    </artifacts>
  </revision>
  <revision name="6.1.26.cloudera.2" evicted="latest-revision" evicted-reason="" downloaded="false" searched="false" conf="compile, master(*), runtime, compile(*), runtime(*), master" position="-1">
    <evicted-by rev="6.1.26.cloudera.4"/>
    <caller organisation="org.apache.hadoop" name="hadoop-common" conf="default, compile, runtime, master" rev="6.1.26.cloudera.2" rev-constraint-default="6.1.26.cloudera.2" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <caller organisation="org.apache.hadoop" name="hadoop-yarn-server-web-proxy" conf="compile, runtime" rev="6.1.26.cloudera.2" rev-constraint-default="6.1.26.cloudera.2" rev-constraint-dynamic="6.1.26.cloudera.4" callerrev="2.6.0-cdh5.4.4"/>
    <artifacts>
    </artifacts>
  </revision>
</module>

The interesting line is this:

<caller organisation="org.apache.hadoop" name="hadoop-hdfs" conf="default, compile, runtime, master" rev="6.1.26" rev-constraint-default="6.1.26" rev-constraint-dynamic="6.1.26" callerrev="2.2.0"/>

I'm not familiar with Ivy, but I'm guessing it means the hadoop-hdfs dependency wants jetty 6.1.26, when everything else wants 6.1.26.cloudera.4 or 6.1.26.cloudera.2

I'm now trying to find the best way to make this work.

update: Found a solution

I don't know if this is a good solution - As I said, I don't know much about Apache Ivy - but it worked for me, and maybe it will work around this issue for you...

I added the following line to sqoop's ivy.xml file, just after the other exclude elements:

<exclude org="org.mortbay.jetty" module="jetty" type="zip" />

I hope somebody will correct me if I'm wrong, but this should prevent Ivy from trying to use the zip type artifact of jetty, and so it will fall-back to something else. In this case, it looks to me that it's using the jar type. Since the jar type is available at the repos ivy is already trying to use, it works.

At least, that's what it looks like to me :)

I also want to point out this patch in Pig's JIRA: https://issues.apache.org/jira/secure/attachment/12707538/PIG-4484-1.patch — Hercynium, Jul 24 '15 at 20:47

Apache sqoop is not compiling

2 Answers2