How value is set in default Linerecordreader in hadoop

Question

Once Jobtracker gets the Splits with getsplits() function of InputFormat class. Then the jobtracker assigns maptasks based on the storage location of the split and maptask calls the createrecordreader() method in InputFormat class which in turn calls linerecordreader class.The initialize function gets the start,end position and nextkeyvalue() sets the key,value. Here is what my query is, Key is set with pos as per below code.But how is the value set.

public boolean  nextKeyValue() throws IOException {
     if (key == null) {
       key = new LongWritable();
     }
     key.set(pos);
     if (value == null) {
       value = new Text();
     }
     int newSize = 0;
     while (pos < end) {
       newSize = in.readLine(value, maxLineLength,
                             Math.max((int)Math.min(Integer.MAX_VALUE, end-pos),maxLineLength))
      if (newSize == 0) {
        break;
      }
      pos += newSize;
      if (newSize < maxLineLength) {
        break;
      }
      // line too long. try again
      LOG.info("Skipped line of size " + newSize + " at pos " + 
               (pos - newSize));
    }
    if (newSize == 0) {
      key = null;
      value = null;
      return false;
    } else {
      return true;
    }
  }

score 1 · Accepted Answer · edited May 23 '17 at 11:43

In nextKeyValue(), while caluculating

newSize = in.readLine(value, maxLineLength,
                             Math.max((int)Math.min(Integer.MAX_VALUE, end-pos),maxLineLength))

Here readLine will populate data into value object. You can refer to readLine implementation here

if (appendLength > 0) {
        str.append(buffer, startPosn, appendLength);
       txtLength += appendLength;
     }

You can refer to this article in SO to understand how actual pass by value is working .

How value is set in default Linerecordreader in hadoop

1 Answers1