MapReduce output as ArrayList

Question

How to call the map reduce method in normal java project and is it possible to return the reducer output as an Arraylist / Hashmap instead of a flat file, and how to Access the mapreduce method from jboss appServer.

Answer 1

Here is a sample program that uses MultipleOutput

    public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int total = 0;
          for (; values.hasNext();) {
            total += values.next().get();
            mos.getCollector("text", reporter).collect(key,
                    new IntWritable(total));
            mos.getCollector("seq", reporter).collect(key,
                    new IntWritable(total));
        }

    }

You would need to create an MultipleOutputs instance in the configure method.

    private MultipleOutputs mos;

    @Override
    public void configure(JobConf job) {

        mos = new MultipleOutputs(job);
    }

In your driver class you need to tell which all inputformats you want to use. Below will generate your output in Text and Sequence file formats.

// Defines additional single text based output 'text' for the job
    MultipleOutputs.addNamedOutput(conf, "text", TextOutputFormat.class,
            Text.class, IntWritable.class);

    // Defines additional sequence-file based output 'sequence' for the job
    MultipleOutputs.addNamedOutput(conf, "seq",
            SequenceFileOutputFormat.class, Text.class, IntWritable.class);

But from what I understood from your question, you basically want to access your mapreduce output from your code. You may download the output file using HDFS API. But better would be to put your data in a Hive table and access using JDBC.