Error: java.lang.NullPointerException error for map reduce program

Question

i map.java class i try to send the filename by this

context.write(new Text(stringWord), new Text(fileName))

recude.java

 import java.io.IOException;

 import org.apache.hadoop.io.*;
 import org.apache.hadoop.mapreduce.*;

 public class Reduce extends Reducer<Text, Text, Text, Text> {

@Override
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
    int sum = 0;
    MapWritable occurenceInFile = new MapWritable();
    for (Text val : values) {
        if(occurenceInFile.containsKey(val)) {
            occurenceInFile.put(val, new IntWritable(1));
        } else {
            IntWritable occurence = (IntWritable) occurenceInFile.get(val);
            occurenceInFile.put(val, new IntWritable(occurence.get() + 1));
        }
        sum += 1;
    }
    String result = key.toString() + " (";
    boolean flag = false;
    for (Writable filenameWritable : occurenceInFile.keySet()) {
        if (flag) {
            result += ", ";
        }
        String filename = ((Text) filenameWritable).toString();
        result += filename + "=" + ((IntWritable) occurenceInFile.get(filenameWritable)).toString();
        flag = true;
    }
    result += ")\nTotal occurence of \"" + key + "\": " + Integer.toString(sum);

    context.write(key, new Text(result));
}
}

error code this error code is show in stderr

Error: java.lang.NullPointerException
at Reduce.reduce(Reduce.java:18)
at Reduce.reduce(Reduce.java:6)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:180)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:656)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:394)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)

i begin to write a wordcount program, but i still do not know how to fix this error.

Look at lines 6 and 18 in your code. This is a word count program? Looks a mess. I'm certain it's possible to write something cleaner. — duffymo, Apr 22 '16 at 18:21
yes this is not whole program, but i think there is no error in my map.java and wordcount.java(Main class) — Adrian Xie, Apr 22 '16 at 18:32
You thought wrong. There is an error. The JVM just made it clear. I'm sure this could and should be written more clearly. I'd recommend a JDK8 stream based version before trying it with Hadoop. If you can't do that, you have little chance with the big hammer. Do something simple first. — duffymo, Apr 22 '16 at 18:58
I like ... There is no error in my code, why is it crashing ? — Romain Hippeau, Apr 22 '16 at 19:19

score 0 · Answer 1 · answered Apr 22 '16 at 18:22

I guess line number #18 is this one:

occurenceInFile.put(val, new IntWritable(occurence.get() + 1));

so it is possible that occurence variable value, which is read on previous line, is null. Another variant is that occurence.get() value is null.

So, it is hard to suggest what can cause it exactly, but you should debug your program and find, why can occurenceInFile.get(val); return null value.

Error: java.lang.NullPointerException error for map reduce program

1 Answers1