1

I got an exception while submitting a mapreduce job from remote system

13/10/28 18:49:52 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/F:/Workspaces/Test/Hadoop/test

My hadoop and mapreduce envirnment is configured on a linux machine. I submit the wordcount job from a local Windows PC as follows:

public static void main(String[] args) throws Exception {

    UserGroupInformation ugi = UserGroupInformation.createRemoteUser("root");

    try {
        ugi.doAs(new PrivilegedExceptionAction<Void>() {

            public Void run() throws Exception {

                JobConf conf = new JobConf(MapReduce.class);
                conf.set("mapred.job.name", "MyApp");
                conf.set("mapred.job.tracker", "192.168.1.149:9001");
                conf.set("fs.default.name","hdfs://192.168.1.149:9000");
                conf.set("hadoop.job.ugi", "root");

                conf.setOutputKeyClass(Text.class);
                conf.setOutputValueClass(IntWritable.class);

                conf.setMapperClass(Map.class);
                conf.setCombinerClass(Reduce.class);
                conf.setReducerClass(Reduce.class);

                conf.setInputFormat(TextInputFormat.class);
                conf.setOutputFormat(TextOutputFormat.class);

                FileInputFormat.setInputPaths(conf, new Path("test"));
                FileOutputFormat.setOutputPath(conf, new Path("test"));

                JobClient.runJob(conf);

                return null;
            }
        });
    } catch (Exception e) {
        e.printStackTrace();
    }
}

where 192.168.1.149 is the hadoop configured linux pc. I started hadoop, mapreduce services there. Also test directory was also created with same java API, it worked. But mapreduce not.

**Please help .. **

C1pher
  • 1,933
  • 6
  • 33
  • 52
Tom Sebastian
  • 3,373
  • 5
  • 29
  • 54
  • Are you able to run the example MapReduce jobs on the machine? http://hadoop.apache.org/docs/r1.2.1/single_node_setup.html under Standalone Operation. – aa8y Oct 28 '13 at 17:02
  • What is the command you are running to execute the job? I Think that may be your problem. What is the path you are giving it when running the job for input? Make sure it is an hdfs path. – Chris Hinshaw Oct 28 '13 at 17:15
  • I'm quite not sure if there is any way you could run a remote job without transferring the jar of your classes to nodes in the cluster. If you think there isn't any I can attempt to post you an answer. – SSaikia_JtheRocker Oct 28 '13 at 19:21
  • I created the "test" floder using java program, in the same way. I could also add files there in that folder. After i run the map reduce program then got that exception "Input path does not exist: file:/F:/Workspaces/Test/Hadoop/test " .This is the folder path of my local PC. I actually intend to specify "test" folder in the hadoop PC "192.168.1.149". How can I? – Tom Sebastian Oct 29 '13 at 04:48

1 Answers1

0

Actually it was my configuration mistake:

I missed mapred.local.dir property in mapred-site.xml 

 

 <property>
    <name>mapred.local.dir</name>
    <value>/usr/local/hadoop-1.0.3/local</value>
 </property>
Tom Sebastian
  • 3,373
  • 5
  • 29
  • 54
  • Folks I tried this it does not work. Getting error like library for your platform... using builtin-java classes where applicable. Then permission issue able to run the job from cluster. – user1927808 Mar 14 '14 at 16:39