1

I am trying to execute a Java class by executing a jar file through java -jar abc.jar but whenever I execute it I get error as :

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

My hadoop-env.sh file in /etc/hadoop/conf directory contains HADOOP_CLASSPATH as:

export HADOOP_CLASSPATH="/home/cloudera/commons-logging-1.1.3.jar:/home/cloudera/hadoop-common-2.4.1.jar:hadoop-core-1.2.1.jar:$HADOOP_CLASSPATH"

In the Java code I have certain imports like: 

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;

and I think it is not able to get the definitions of these at run time. what is causing these errors and how do I resolve this issue?

NOTE: I am new to hadoop and as such I do not know what and how to set different environment variables needed. Any example will be helpful for me.

UPDATE: After I executed java -cp $HADOOP_CLASSPATH:abc.jar:/home/extra_jars/* mypackage.classname I get error as java.io.IOException: Mkdirs failed to create /user/example

In my code I am executing fs.copyFromLocalFile(false,true,sourcePath,targetPath); and the targetPath is /user/example/ex1.csv. I am unable to understand why this error is coming even though my /user/example has full red-write permissions.

user2966197
  • 2,793
  • 10
  • 45
  • 77
  • try to include the classpath: `java -cp $HADOOP_CLASSPATH -jar abc.jar` – bachr Oct 27 '14 at 11:16
  • when I execute `java -cp $HADOOP_CLASSPATH -jar abc.jar `I get error as `Error: Could not find or load main class abc.jar` – user2966197 Oct 27 '14 at 11:20
  • what about `java -cp $HADOOP_CLASSPATH:abc.jar your.package.YourClassName` – bachr Oct 27 '14 at 11:23
  • @Arbi the previous error of `java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration` – user2966197 Oct 27 '14 at 11:28
  • The error `java.io.IOException: Mkdirs failed to create /user/example` usually means that the user you are running the program as does not have enough privileges to create a directory `/user/example`. If this is a CDH or HDP cluster try running the program as `hdfs` user or if this is a apache hadoop cluster try running as superuser (usually `hadoop` user). – Ashrith Oct 27 '14 at 16:26
  • @AshrithM I am running a CDH cluster. How do I run the program as `hdfs` user? I mean how do I switch user from command line? – user2966197 Oct 27 '14 at 20:16
  • You could use sudo like this `sudo -u hdfs java -cp $HADOOP_CLASSPATH:your.jar yourpackage.yourclassname` – Ashrith Oct 27 '14 at 20:37
  • @AshrithM I tried this way but still same `Mkdirs failed to create` error – user2966197 Oct 27 '14 at 20:50
  • @AshrithM I found this post on so http://stackoverflow.com/questions/10522835/hadoop-java-io-ioexception-mkdirs-failed-to-create-some-path but couldn't understand the accepted answer here – user2966197 Oct 27 '14 at 20:55
  • May be this http://stackoverflow.com/questions/10522835/hadoop-java-io-ioexception-mkdirs-failed-to-create-some-path might shed more light – Ashrith Oct 27 '14 at 20:55
  • @AshrithM The accepted answer says that the file is on local disk but on my local disk there is no directory `/user/example`. This exists in `hdfs` only – user2966197 Oct 27 '14 at 20:58
  • I created the `/user/example` directory on my local system and executed the jar again and it ran successfully. It is not writing to the `hdfs` even though the code uses `copyFromLocal()` which writes to hdfs – user2966197 Oct 27 '14 at 21:24
  • Do you have the folder `/user/example` created in HDFS? Or are you trying to create them in the code? If in code, as someone pointed out, it could be an access problem. Also check that you have the datanode and namenode directories set in config. – Lauri Peltonen Oct 28 '14 at 07:15
  • @LauriPeltonen Yes I already have created the the `/user/exampl` in HDFS. The namenode I using is same as mentioned in the `core-site.xml` file. – user2966197 Oct 28 '14 at 15:29
  • Namenode and datanode folders should be defined in hdfs-site.xml. Can you show us that file. – Lauri Peltonen Oct 29 '14 at 06:40

0 Answers0