spring data hive integration with hive template

Question

I am trying to use spring data hadoop to integrate hive into my application and running into some issues. First thing I am not sure about is <hdp:hive-server host="some-other-host" port="10001" /> is this to connect to an existing hive server or to something like create a new hive server to then be able to connect to it. Secondly my configuration does not throws any errors so it does seems ok and even the hiveTemplate autowiring works fine too but when I execute a query I dont seem to get any response back. The application sort of gets stuck at that point.

here is the configuration

<hive-client-factory host="${hive-${env}.server}" port="${hive-${env}.port}" />

<hive-template />

and here is how im using it

log.debug("before hive query");

for(String result : hiveTemplate.query("show tables;")){
    log.debug("=> " + result);
}

log.debug("after hive query");

all I see in log output is before hive query .. nothing happens after that. I would appreciate any help. Any ideas what I could be doing wrong.

score 0 · Answer 1 · answered Aug 22 '14 at 04:48

Try 10000 as the port number.
Usually Thrift server is deployed with port number 10000. Check If your installation is using HiveServer2 or HiveServer. I was able to get Spring Batch workflows to work with HiveServer, but I have no yet succeeded with HiveServer2.
Ensure that you have HiveServer/HiveServer is up and running before ou run your program

score 0 · Answer 2 · answered Jul 04 '17 at 03:45

If we are using with spring,please make sure all your dependency is compatible with spring version.After that give read and write permission using below commands.

hadoop fs -mkdir /tmp
hadoop fs -chmod a+w /tmp
hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -chmod a+w /user/hive/warehouse

I have used following version of dependency in pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <artifactId>spring-hadoop-samples-hive</artifactId>

    <name>Spring Hadoop Samples - Hive</name>

    <parent>
        <groupId>org.springframework.samples</groupId>
        <artifactId>spring-hadoop-samples</artifactId>
        <version>1.0.0.BUILD-SNAPSHOT</version>
        <relativePath>../parent/pom.xml</relativePath>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spring.hadoop.version>2.3.0.M1</spring.hadoop.version>
        <hadoop.version>2.7.1</hadoop.version>
        <hive.version>1.2.1</hive.version>
        <!-- <hive.version>2.1.1</hive.version> -->
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-hadoop</artifactId>
            <version>${spring.hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework</groupId>
                    <artifactId>spring-context-support</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-jdbc</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-tx</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-metastore</artifactId>
            <version>${hive.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-service</artifactId>
            <version>${hive.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libfb303</artifactId>
            <version>0.9.1</version>
        </dependency>

        <!-- runtime Hive deps start -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-common</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-shims</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-serde</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-contrib</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <!-- runtime Hive deps end -->

        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy</artifactId>
            <version>1.8.5</version>
            <scope>runtime</scope>
        </dependency>

    </dependencies>

    <repositories>
        <repository>
            <id>spring-milestone</id>
            <url>http://repo.spring.io/libs-milestone</url>
        </repository>
    </repositories>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>appassembler-maven-plugin</artifactId>
                <version>1.2.2</version>
                <configuration>
                    <repositoryLayout>flat</repositoryLayout>
                    <configurationSourceDirectory>src/main/config</configurationSourceDirectory>
                    <copyConfigurationDirectory>true</copyConfigurationDirectory>
                    <!-- Extra JVM arguments that will be included in the bin scripts -->
                    <extraJvmArguments>-Xms512m -Xmx1024m -Dhive.version=${hive.version}</extraJvmArguments>
                    <programs>
                        <program>
                            <mainClass>org.springframework.samples.hadoop.hive.HiveApp</mainClass>
                            <name>hiveApp</name>
                        </program>
                        <program>
                            <mainClass>org.springframework.samples.hadoop.hive.HiveClientApp</mainClass>
                            <name>hiveClientApp</name>
                        </program>
                        <program>
                            <mainClass>org.springframework.samples.hadoop.hive.HiveAppWithApacheLogs</mainClass>
                            <name>hiveAppWithApacheLogs</name>
                        </program>
                    </programs>
                </configuration>
                <executions>
                    <execution>
                        <id>package</id>
                        <goals>
                            <goal>assemble</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <executions>
                    <execution>
                        <id>config</id>
                        <phase>package</phase>
                        <configuration>
                            <tasks>
                                <copy todir="target/appassembler/data">
                                    <fileset dir="data"/>
                                </copy>
                            </tasks>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

spring data hive integration with hive template

2 Answers2