MRUnit - Not Working as expected

Question

I am having only mapper class that creates files for bulk loading into HBase and I have written a MRUnit for Unit testing.

Though the expected result and the retruned results are same, the MRUnit is failing with the message "Missing expected output".

Expected output: (4b 65 79 31, {"totalColumns":1,"families":{"default":[{"timestamp":9223372036854775807,"qualifier":"default","vlen":6}]},"row":"Key1"})

Actual output: (4b 65 79 31, {"totalColumns":1,"families":{"default":[{"timestamp":9223372036854775807,"qualifier":"default","vlen":6}]},"row":"Key1"})

Where am I going wrong ? Is it problem with ImmutableBytesWritable Or Put ?

Output key class type : ImmutableBytesWritable ,Output value class type: Put.... MRUnit version: 0.9.0 — Suresh, Feb 14 '13 at 10:56

score 9 · Accepted Answer · answered Feb 14 '13 at 12:48

So your failure is the fact that MRUnit is using the equals and hashCode methods of the objects to compare the expected and actual output objects (a HashMap is used in the org.apache.hadoop.mrunit.TestDriver.buildPositionMap(List<Pair<K2, V2>>) method):

ImmutableBytesWritable - Does implement an equals and hashCode method
Put - Doesn't implement an equals or hashCode method

So the Put is what's causing the problem. The only way around this is to manually check the driver output (use the driver run method rather than runTest) and compare the result List to what you're expecting using the objects compareTo method (which does exist for both classes).

score 0 · Answer 2 · answered Jan 07 '16 at 06:36

I had this problem and solved it by setting a Comparator for Key and Value. Code below for a BytesWritable value.

    ReduceDriver reduceDrive = ReduceDriver.newReduceDriver(<<ReducerInstance>>);

    reduceDriver.setValueComparator(new Comparator<BytesWritable>(){
        @Override
        public int compare(BytesWritable o1, BytesWritable o2) {
            return o1.compareTo(o2);
        }
    });

MRUnit - Not Working as expected

2 Answers2