Include Third Party Jars in Hadoop

Question

I am new to Hadoop. I have added Gson API to my MapReducing Program. When I am running the program getting;

Error: java.lang.ClassNotFoundException: com.google.gson.Gson

Can anybody suggest me to how to add Third Party Libraries to Hadoop?

Check out this question http://stackoverflow.com/questions/11345923/how-to-import-org-apache-java-dependancies — Dan Ciborowski - MSFT, May 02 '13 at 14:05
Check my answer here, I have explained all the available options to fix this issue here: http://stackoverflow.com/a/36227260/1766402 — Isaiah4110, Mar 27 '16 at 00:37

score 3 · Answer 1 · answered May 02 '13 at 14:46

3

Be sure to add any dependencies to both the HADOOP_CLASSPATH and -libjars upon submitting a job like in the following examples:

Use the following to add all the jar dependencies from current and lib directories:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:`echo *.jar`:`echo lib/*.jar | sed 's/ /:/g'`

Bear in mind that when starting a job through hadoop jar you'll need to also pass it the jars of any dependencies through use of -libjars. I like to use:

hadoop jar <jar> <class> -libjars `echo ./lib/*.jar | sed 's/ /,/g'` [args...]

NOTE: The sed commands require a different delimiter character; the HADOOP_CLASSPATH is : separated and the -libjars need to be , separated.

answered May 02 '13 at 14:46

Quetzalcoatl

3,037
2
18
27

I tried your advise by doing like.. bin/hadoop jar /home/Projects/MyMaprunner/dist/progenMaprunner.jar -libjars /home/hadoopJar/gson-2.2.2.jar /myData /output arg1 arg2 got exception as Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/MyDesktop/-libjars – sri_sankl May 03 '13 at 07:04
Check the answer posted here: http://stackoverflow.com/questions/6890087/problem-with-libjars-in-hadoop – Vladimir Kroz Nov 06 '13 at 03:27

score 0 · Answer 2 · answered Aug 18 '15 at 13:07

Add the Jar in HADOOP_CLASSPATH

    vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Add last line

export HADOOP_CLASSPATH=/root/hadoop/extrajars/java-json.jar:$HADOOP_CLASSPATH

"/root/hadoop/extrajars/java-json.jar" is path on linux box itself and not on HDFS

Restart the hadoop

Command

hadoop classpath

Should show the jar in classpath

Now run MR job as usual

hadoop jar <MR-program jar> <MR Program class> <input dir> <output dir>

It will use the file from as expected.

Include Third Party Jars in Hadoop

2 Answers2

Linked