14

I'm running a Hadoop job using Hive actually that is supposed to uniq lines in many text files. In the reduce step, it chooses the most recently timestamped record for each key.

Does Hadoop guarantee that every record with the same key, output by the map step, will go to a single reducer, even if many reducers are running across a cluster?

I worry that the mapper output might be split after the shuffle happens in the middle of a set of records with the same key.

Gyan Veda
  • 6,309
  • 11
  • 41
  • 66
samg
  • 3,496
  • 1
  • 25
  • 26

3 Answers3

14

All values for a key are sent to the same reducer. See this Yahoo! tutorial for more discussion.

This behavior is determined by the partitioner, and might not be true if you use a partitioner other than the default.

Karl Anderson
  • 1,798
  • 11
  • 18
  • actually i am not sure of this. See http://stackoverflow.com/questions/26693034/hadoop-strange-behaviour-reduce-function-doesnt-get-all-values-for-a-key . I didn't modify the partitioner in my program. – Madrugada Nov 01 '14 at 23:50
5

Actually, no! You could create a Partitioner that sent the same key to a different reducer each time the getPartition is called. It's just not generally a good idea for most applications.

Bkkbrad
  • 3,087
  • 24
  • 30
3

Yes, Hadoop does guarantee that all keys that are the same will go to the same Reducer. This is achieved using a Partition function which buckets the keys using a hash function.

For more information on the Partitioning process take a look at this: Partitioning Data

It specifically talks about how different mappers that process the same key ensure that all keys of a given value end up in the same partition, and thus are processed by the same reducer.

Binary Nerd
  • 13,872
  • 4
  • 42
  • 44