0

I have a question. I want to iterate through reduce result (Reduce keys of type Text)in a mapreduce program. how can I do this? can I convert the result into a ResultSet? Here is My code:

 protected void setup(Mapper.Context context) throws IOException, InterruptedException {
        sourceColumn = ByteBufferUtil.bytes(context.getConfiguration().get(CONF_COLUMN_NAME));
    }

    public void map(ByteBuffer key, SortedMap<ByteBuffer, IColumn> columns, Context context) throws IOException, InterruptedException
    {
       for (IColumn col: columns.values()){
        String value = ByteBufferUtil.string((col.value()));
        StringTokenizer itr = new StringTokenizer(value);
        while (itr.hasMoreTokens()) {
            word.set(new Text(itr.nextToken()));
            context.write(word, one);
        }           
    }

 }


protected void setup(Reducer.Context context) throws IOException, InterruptedException
    {
        // The row key is the name of the column from which we read the text
        outputKey = ByteBufferUtil.bytes(context.getConfiguration().get(CONF_COLUMN_NAME));
    }

    public void reduce(Text word, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException
    {
        int sum = 0;
        for (IntWritable val : values) {
            sum += val.get();
        }
        context.write(outputKey, Collections.singletonList(getMutation(word, sum)));

        Text date = context.getCurrentKey();
    }

now I want to iterate over date. for example like a resultset or an array..

Anse danesh
  • 633
  • 3
  • 7
  • 17
  • Can you show us the code you use to retrieve the keys and results and tell us what environment you are using? – Lyuben Todorov Sep 26 '13 at 08:39
  • You can write the result into a sequence file and use that as input for another M/R job... – Remus Rusanu Sep 26 '13 at 08:47
  • I edit my question and add the codes.. can you help me I want to iterate through date.. – Anse danesh Sep 26 '13 at 09:25
  • Remus Rusanu- How can I write these results into a sequence file? you mean output to a system file? – Anse danesh Sep 26 '13 at 09:27
  • 1
    You control the output format from the [`mapred.output.format.class`](http://wiki.apache.org/hadoop/JobConfFile) job property. [`SequenceFileOutputFormat`](http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapred/SequenceFileOutputFormat.html) is the class for sequence files. See http://stackoverflow.com/a/5382286/105929 , see http://stackoverflow.com/questions/5700068/merge-output-files-after-reduce-phase – Remus Rusanu Sep 26 '13 at 10:07
  • thanks for your clear reply.. but I want to return date as a variable that can be iterate.. like an array or a resultset... is the above property help me to do this? – Anse danesh Sep 26 '13 at 15:32
  • 1
    You can't iterate in memory. The output may be (*way*) larger than your available Java heap size. Use an intermediate file and a second M/R job. – Remus Rusanu Sep 27 '13 at 08:57

0 Answers0