5

I was trying to set up a Maven project that will contain user defined functions (UDFs) that I'd like to use in my Hive queries. I started with a Maven project containing no source files, and the following POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>exp</groupId>
    <artifactId>HiveUdfTestProject</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>0.14.0</version>
        </dependency>
    </dependencies>
</project>

When I tried to build the project, I get the following error:

Failed to execute goal on project HiveUdfTestProject: Could not resolve dependencies for project exp:HiveUdfTestProject:jar:1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.calcite:calcite-core:jar:0.9.2-incubating-SNAPSHOT, org.apache.calcite:calcite-avatica:jar:0.9.2-incubating-SNAPSHOT: Could not find artifact org.apache.calcite:calcite-core:jar:0.9.2-incubating-SNAPSHOT -> [Help 1]

Julian Hyde
  • 1,239
  • 7
  • 10
Steven Magana-Zook
  • 2,751
  • 27
  • 41

3 Answers3

3

I found the calcite-core-incubating jar in the maven central repository (but not the incubating-snapshot version) required by the hive-exec 0.14.0 dependency.

Adding the calcite-core from maven central got rid of the original error, and introduced a new missing dependency "pentaho-aggdesigner-algorithm" which I found on ConJars.

Adding the conjars repo and the pentaho dependency made a new missing dependency appear "org.apache.calcite:calcite-avatica:jar:0.9.2-incubating-SNAPSHOT" whose incubating (but not snapshot) dependency was available in the maven central repo.

Adding the calcite-avatica dependency to the POM made the empty project build successfully at last.

Here is the final POM needed to make a project intended for Hive UDFs build:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>exp</groupId>
    <artifactId>HiveUdfTestProject</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>
    <repositories>
        <repository>
            <id>conjars.org</id>
            <url>http://conjars.org/repo</url>
        </repository>
    </repositories>
    <dependencies>
        <!-- From Maven Central -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>0.14.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.calcite</groupId>
            <artifactId>calcite-core</artifactId>
            <version>0.9.2-incubating</version>
        </dependency>
        <dependency>
            <groupId>org.apache.calcite</groupId>
            <artifactId>calcite-avatica</artifactId>
            <version>0.9.2-incubating</version>
        </dependency>

        <!-- From conjars -->
        <dependency>
            <groupId>org.pentaho</groupId>
            <artifactId>pentaho-aggdesigner-algorithm</artifactId>
            <version>5.1.3-jhyde</version>
        </dependency>
    </dependencies>
</project>

Once the empty project built, I tried integrating the POM settings into a larger existing Maven project and saw errors about calcite-core specifically looking for the snapshot version. To get past this, I changed the hive-exec dependency to look like this:

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-exec</artifactId>
    <version>0.14.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.calcite</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>
Steven Magana-Zook
  • 2,751
  • 27
  • 41
2

I explicitly included the calcite-core and calcite-avatica projects as dependencies, and my project (which also includes the Hive 14 dependencies), no longer failed with the 'artifacts could not be resolved error'

   <dependency>
           <groupId>org.apache.calcite</groupId>
           <artifactId>calcite-core</artifactId>
           <version>1.0.0-incubating</version>
   </dependency>
   <dependency>
           <groupId>org.apache.calcite</groupId>
           <artifactId>calcite-avatica</artifactId>
           <version>1.0.0-incubating</version>
   </dependency>

From what I can tell, this is an open issue with Hive 14. See https://issues.apache.org/jira/browse/HIVE-8906 for more info.

ryanbwork
  • 2,123
  • 12
  • 12
0

I've resolved the same issue by adding below dependencies to /ql/pom.xml org.pentaho pentaho-aggdesigner-algorithm 5.1.3-jhyde 

<dependency>
  <groupId>eigenbase</groupId>
  <artifactId>eigenbase-properties</artifactId>
  <version>1.1.4</version>
</dependency>

<dependency>
  <groupId>net.hydromatic</groupId>
  <artifactId>linq4j</artifactId>
  <version>0.4</version>
</dependency>

and below repository to /pom.xml under positories

<repository>
      <id>conjars</id>
      <name>Concurrent Conjars repository</name>
      <url>http://conjars.org/repo</url>
      <layout>default</layout>
    </repository>
Niranjan Sarvi
  • 899
  • 7
  • 9