reducer with iterator error

Question

I have a reducer in hadoop with this code:

public void reduce(Text key, Iterator<Text> values, Context context) throws IOException, InterruptedException {
/*some code*/
String followers = ""; 
  while(values.hasNext()){
        followers = followers + values.next().toString() + ",";
  }
/*some code*/
}

I want to make a list of followers for a certain node but when I run it I am getting this:

Error: java.lang.ArrayIndexOutOfBoundsException: 1

If values is type Iterable then I don't have a problem but why is this happening?

thanks in advance.

I would guess that you're hitting the [max string length](http://stackoverflow.com/questions/1179983/how-many-characters-can-a-java-string-have), but I can't tell without knowing how many values are in the iterator/ how big the values are — lucasvw, Jul 01 '16 at 21:34
I am trying a very small input so I do not think I am hitting max string length. — Suzy Tros, Jul 01 '16 at 21:37
Not an answer to your question, but a few side points: 1) You can do `followers += values...` 2) You can do `...values.next() + ",";`, since `toString()` is called implicitly 3) You should usually use a `StringBuilder` when concatenating strings in a loop. — shmosel, Jul 01 '16 at 21:44

Binary Nerd · Accepted Answer · 2016-07-02T09:20:38.770

0

If you're extending org.apache.hadoop.mapreduce.Reducer (Java Docs) the method signature you've used for reduce() is incorrect. When you implement the reduce() method, you are overriding the implementation in the Reduce class.

You have:

reduce(Text key, Iterator<Text> values, Context context)

It should be:

reduce(Text key, Iterable<Text> values, Context context)

Note the difference between Iterator and Iterable. This probably means your reduce isnt actually being called.

You can add the @Override annotation to the method to help prevent this.

edited Jul 02 '16 at 09:20

answered Jul 02 '16 at 09:13

Binary Nerd

13,872
4
42
44

but when should I use iterator and when iterable in hadoop? – Suzy Tros Jul 02 '16 at 09:23
Your method signature must use Iterable, otherwise it wont actually get called. – Binary Nerd Jul 02 '16 at 09:24
i'm not sure i understand. We're talking about the method signature, that signature is fixed. If something is `Iterable` you can easily loop using `for (Text t : values) { }`. If you ever see `Iterator` in a reduce method signature, they've got it wrong. – Binary Nerd Jul 02 '16 at 09:29

reducer with iterator error

1 Answers1