Driver class compilation error - hadoop Mapreduce

Question

I have 3 .java file

1) Mapper.java
2) Reducer.java
3) Driver.java

I am trying to compile hadoop mapreduce program at command line using Driver class but it is showing below error

Driver.java:39: error: cannot find symbol
        job.setMapperClass(Mapper.class);
                           ^
  symbol:   class Mapper
  location: class Driver
Driver.java:40: error: cannot find symbol
        job.setReducerClass(Reducer.class);

How can I solve above error.Below is run method in Driver class

public boolean runnerParsing(String inputPath, String outputPath) throws IOException, ClassNotFoundException, InterruptedException {
         Configuration conf = new Configuration();

        Job job = new Job(conf, "Parsing");
        job.setJarByClass(Driver.class);


        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        job.setMapperClass(Mapper.class);
        job.setReducerClass(Reducer.class);
        //job.setNumReduceTasks(0);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.addInputPath(job, new Path(inputPath));
        FileOutputFormat.setOutputPath(job, new Path(outputPath));

        return job.waitForCompletion(true);

     }

score 1 · Answer 1 · edited Oct 13 '16 at 19:31

You will need to compile all java files as below:

javac -classpath /usr/local/hadoop/hadoop-core-1.2.1.jar -d compiled_classes Driver.java Mapper.java Reducer.java

Please note that your classpath value may change slightly depending on how you install Hadoop.

If you need further help, please have a look at this article which might help you: http://www.bigdatatutes.com/getting-started-with-big-data/

score 0 · Answer 2 · answered Oct 26 '14 at 16:37

0

This is what i think:

You don't have Mapper/Reducer stored in Mapper.java/Reducer.java.
You have mapper/reducer class in another package

Can you add your mapper and reducer class as well to question?

answered Oct 26 '14 at 16:37

SMA

36,381
8
49
73

Oh I am really I found solution for this but I forgot to remove question.I compiled all of the files together and it worked. – user2895589 Oct 26 '14 at 20:42

score 0 · Answer 3 · answered Sep 15 '19 at 16:15

In Chapter 3 of “Big Data with Hadoop” byGarry Tukington and Gabriele Modena, there is the example of pattern Top N where, in the driver code of TopTenHashTag class, there is this instruction with a reference to another java class :

job.setJarByClass(HashTagCount.class);

In my PC, both files, HashTagCount.java and TopTenHashTag.java are in: /home/hduser/playground/src

My compilation command is the following (and has worked for me) :

javac -classpath $HADOOP_HOME/share/hadoop/common/lib/activation-1.1.jar:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.1.jar:$HADOOP_HOME/share/hadoop/common/lib/:/usr/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/ -d playground/classes7 playground/src/TopTenHashTag.java \ playground/src/HashTagCount.java

This is the command to create the .jar file : jar -cvf playground/TopTenHashTag.jar -C playground/classes7/ .

and finally this is the command to start the mapreduce job :

hadoop jar /home/hduser/playground/TopTenHashTag.jar com.learninghadoop2.mapreduce.TopTenHashTag /user/hduser/inxYZ/ outHashXYZ

Driver class compilation error - hadoop Mapreduce

3 Answers3

Linked