0

Using Eclipse I have built a MapReduce app. that uses also HCatalog and Hive jar files. This app was developed in Eclipse project on a local Hadoop installation. All neccessary jar files were simply included in Eclipse project from directories where Hadoop and Hive are installed on local node.

Now I need to run this app. on multi-node Hadoop installed elsewhere. Please advise on the best way tp package all neccessary jars in a single app. Or should I set CLASSPATH on the node where I plan to run my app? It would be nice to build a single self-sufficient jar right in Eclipse.

Thanks!

DarqMoth
  • 603
  • 1
  • 13
  • 31
  • The q. is on hold for some reason so I'll leave this as a comment. There are several ways to do that - take a look here http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/ – Arnon Rotem-Gal-Oz Mar 22 '14 at 15:22

1 Answers1

1

Maven is a very good build tool - http://maven.apache.org/ It can package all your classes in a jar and can also add all the needed jar libraries inside it.

You can convert your existing project to Maven, by right clicking the project and choose menu Configure > Convert to maven project. And then add the needed dependencies in pom.xml.

An example dependencies would look like.

   <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.0.0-cdh4.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-auth</artifactId>
            <version>2.0.0-cdh4.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.0.0-cdh4.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.0.0-mr1-cdh4.0.1</version>
        </dependency>

Also see http://hadoopi.wordpress.com/2013/05/25/setup-maven-project-for-hadoop-in-5mn/

Jay
  • 9,189
  • 12
  • 56
  • 96
  • How Maven can help me to move my app with all jars to another machine? – DarqMoth Mar 21 '14 at 13:39
  • Maven helps to package all your needed class and libraries. For copying files to remove server using maven check this http://stackoverflow.com/questions/909867/maven-copy-local-file-to-remote-server-using-ssh – Jay Mar 21 '14 at 13:47
  • Thanks for the links. After conversion to Maven project Eclipse created a vanila pom.xml. Now I need to manually insert all dependenies in this pom.xml, correct? Looks like in this case Maven does not help much to package an app from Eclipse... – DarqMoth Mar 21 '14 at 14:29