Text not converting to String Hadoop Java

Question

I am trying to convert Text to String in my reduce function but its not working. I tried the same logic in Map function and it worked perfectly, but when I tried to apply this logic in my reduce function it is giving error: java.lang.ArrayIndexOutOfBoundsException 1

My Map code is like this

public static class OutDegreeMapper2 
    extends Mapper<Object, Text, Text, Text>
{

    private Text word = new Text();
    private Text word2 = new Text();

    public void map(Object key, Text value, Context context
                        ) throws IOException, InterruptedException 
    {
        String oneLine = value.toString();
        String[] parts = oneLine.split("\t");
        word.set(parts[0]);
        String join = parts[1]+",from2";
        word2.set(join);

        context.write(word, word2);
    }
}

My reduce function is like this

public static class OutDegreeReducer 
    extends Reducer<Text,Text,Text,Text> 
{
    private Text word = new Text();
    String merge ="";
    public void reduce(Text key, Iterable<Text> values, 
                                Context context
                        ) throws IOException, InterruptedException 
    {

        for(Text val:values)
        {

            String[] x = val.toString().split(",");

            if(x[1].contains("from2")){
                merge+= x[0];
            }

        }
        word.set(merge);
        context.write(key, word);
    }
}

Kindly tell me why split is working in map function but not in reducer?

need stacktrace, ArrayIndexOutOfBoundsException: This would mean you are trying accessing an index in the array that does not exists. Without more stacktrace, I cannot help here. — Roger, Apr 27 '17 at 22:38
Can you kindly tell me what do you mean by stacktrace? I am quite new to programming so don't know about this much. — Hasan Iqbal, Apr 27 '17 at 22:43
Strange this is split function is working perfectly in Map function but it is not working in reducer function. I even checked it with val.toString().contains("from2") to validate and it does not even pick it. My output shows empty values against keys. — Hasan Iqbal, Apr 27 '17 at 22:45
Please [edit] your question with the input text you are providing — OneCricketeer, Apr 27 '17 at 22:48
Stacktrace => http://stackoverflow.com/questions/3988788/what-is-a-stack-trace-and-how-can-i-use-it-to-debug-my-application-errors If you are not comfortable with Java and debugging it, then I wouldn't suggest learning MapReduce — OneCricketeer, Apr 27 '17 at 22:49

score 0 · Answer 1 · answered Apr 27 '17 at 22:43

0

Very likely here

String[] parts = oneLine.split("\t");
word.set(parts[0]);
String join = parts[1]+",from2";

or here

String[] x = val.toString().split(",");

if(x[1].contains("from2")){
    merge+= x[0];
}

when read x[1] or parts[1] throws the ArrayIndexOutOfBoundsException because there is no , and \t inside the string.

I suggest to check the size of the array before access the element 1.

Looking at the stacktrace you should be able to understand where is throwing the exception.

answered Apr 27 '17 at 22:43

freedev

25,946
8
108
125

Your first mentioned part is working perfectly, i crossed checked it by removing the if condition in reducer. And it printed from2 against keys. But when I try to split the Text by converting it into string or try to check function val.toString().contains, it doesnt work. strange thing as it works in map fucntion but not in reduce. – Hasan Iqbal Apr 27 '17 at 22:51
You say there is an `ArrayIndexOutOfBoundsException` right? – freedev Apr 27 '17 at 22:52
if the array returned by `split` is smaller that 2 elements an `ArrayIndexOutOfBoundsException` is thrown. I only see 2 points in your code where this can happens. Just add in the statement `if (x[1].contains("from2"))` a check like `if (x.length > 1 && x[1].contains("from2"))`. do the same for the other `parts[1]` – freedev Apr 27 '17 at 22:57
When I added x[0] to the code directly, it worked. Output shows from1 against keys. But when I try to use val.toString.contains("from2") it doesn't works. same case with split. Split and contains don't work here – Hasan Iqbal Apr 27 '17 at 23:02
Waiting for you reply sir :( – Hasan Iqbal Apr 27 '17 at 23:19

score 0 · Answer 2 · answered Apr 28 '17 at 00:22

0

Instead of

  if(x.length() > 1 && x[1].contains("from2")){
        merge+= x[0];
    }

Do this:

if(x.length() > 1 && x[1].contains("from2")){
    merge+= x[0];
}

answered Apr 28 '17 at 00:22

Roger

2,823
3
25
32

I tried to use x.length() function and it gives following error BFS.java:110: error: cannot find symbol int j=x.length(); – Hasan Iqbal Apr 28 '17 at 05:36

Text not converting to String Hadoop Java

2 Answers2