8

I'm trying to use MRunit to test my sortComparatorClass. It seems MRunit should be able to do this with the setKeyOrderComparator method, but when I run the mapReduceDriver it is not calling the compare() method of SortComparator class.
Pretty sure I'm doing something wrong with MRunit API.

Here's my code for the unit test:

public class UnitTests {

private static transient Log log = LogFactory.getLog(UnitTests.class);

MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver;
MapDriver<Text, Text, Text, Text> mapDriver;
ReduceDriver<Text, Text, Text, Text> reduceDriver;

@Before
public void setUp() throws InterruptedException, IOException {
    mapDriver = new MapDriver<Text, Text, Text, Text>();
    mapDriver.setMapper(new TestMapper());
    reduceDriver = new ReduceDriver<Text, Text, Text, Text>();
    reduceDriver.setReducer(new TestReducer());
    mapReduceDriver = new MapReduceDriver(new TestMapper(), new TestReducer());
    mapReduceDriver.setKeyOrderComparator(new TestSortCompartor());
}

@Test
public void testSort() throws IOException {

    Text inputKey1 = new Text("def");
    Text inputKey2 = new Text("abc");
    Text inputValue = new Text("BlahBlahBlah");
    mapReduceDriver.addInput(inputKey1, inputValue);
    mapReduceDriver.addInput(inputKey2, inputValue);

    List<Pair<Text, Text>> output = mapReduceDriver.run();
    log.info("Got output of size "+output.size()+" with first pair = "+output.get(0).toString());
}
}

And here's my test sortComparator:

public class TestSortCompartor extends WritableComparator{
private static transient Log log = LogFactory.getLog(TestSortCompartor.class);

public TestSortCompartor() {
    super(Text.class, true);
}

@SuppressWarnings("rawtypes")
@Override
public int compare(WritableComparable w1, WritableComparable w2) {
    log.info("calling compare with key1 = "+w1.toString()+" and key2 "+w2.toString());
    return w1.compareTo(w2) * -1;
}
}

When I run the test I get this output:

INFO 2014-01-13 09:34:27,362 [main] (com.gradientx.gxetl.testMR.UnitTests:53) - 
Got output of size 2 with first pair = (abc, BlahBlahBlah)

But there's no output from the sortComparator class - and it's not ordering the keys in reverse, so I know my comparator() method is not being called.

Can anyone advise what am I doing wrong? Is it possible to use MRunit to test my own comparator class? Is there a better way to make a unit test for a custom comparator class?

FYI, here are the relevant dependencies from my Pom:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.0.0-mr1-cdh4.4.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.0.0-mr1-cdh4.4.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.mrunit</groupId>
    <artifactId>mrunit</artifactId>
    <version>1.0.0</version>
    <classifier>hadoop2</classifier>
</dependency>
Manjunath Ballur
  • 6,287
  • 3
  • 37
  • 48
alexP_Keaton
  • 349
  • 5
  • 18

0 Answers0