Use MRUnit and AVRO together

Question

I have created a Mapper & Reducer which use AVRO for input, map-output en reduce output. When creating a MRUnit test i get the following stacktrace:

java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
at org.apache.hadoop.mrunit.mock.MockOutputCollector.deepCopy(MockOutputCollector.java:74)
at org.apache.hadoop.mrunit.mock.MockOutputCollector.collect(MockOutputCollector.java:110)
at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper$MockMapContext.write(MockMapContextWrapper.java:119)
at org.apache.avro.mapreduce.AvroMapper.writePair(AvroMapper.java:22)
at com.bol.searchrank.phase.day.DayMapper.doMap(DayMapper.java:29)
at com.bol.searchrank.phase.day.DayMapper.doMap(DayMapper.java:1)
at org.apache.avro.mapreduce.AvroMapper.map(AvroMapper.java:16)
at org.apache.avro.mapreduce.AvroMapper.map(AvroMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:200)
at org.apache.hadoop.mrunit.mapreduce.MapReduceDriver.run(MapReduceDriver.java:207)
at com.bol.searchrank.phase.day.DayMapReduceTest.shouldProduceAndCountTerms(DayMapReduceTest.java:39)

The driver is initialized as follows (i have created a Avro MapReduce API implementation):

    driver = new MapReduceDriver<AvroWrapper<Pair<Utf8, LiveTrackingLine>>, NullWritable, AvroKey<Utf8>, AvroValue<Product>, AvroWrapper<Pair<Utf8, Product>>, NullWritable>().withMapper(new DayMapper()).withReducer(new DayReducer());

Adding a configuration object with io.serialization won't help:

    Configuration configuration = new Configuration();
    configuration.setStrings("io.serializations", new String[] {
        AvroSerialization.class.getName()
    });
    driver = new MapReduceDriver<AvroWrapper<Pair<Utf8, LiveTrackingLine>>, NullWritable, AvroKey<Utf8>, AvroValue<Product>, AvroWrapper<Pair<Utf8, Product>>, NullWritable>().withMapper(new DayMapper()).withReducer(new DayReducer()).withConfiguration(configuration);

I use Hadoop & MRUnit 0.20.2-cdh3u2 from Cloudera and Avro MapRed 1.6.3.

What version of mrunit are you using, and are you configuring the Configuration `io.serializations` before you create the driver? — Chris White, Apr 12 '12 at 12:03
The configurion is created before the driver, i updated the statements in accordance. Also added the version numbers. — Salandur, Apr 13 '12 at 08:53
Is there more code missing here - i get a compile error for your MapReduceDriver contructor line: `Bound mismatch: The type AvroKey is not a valid substitute for the bounded parameter of the type MapReduceDriver`. I can get a simple unit test of the serialization factory working though (without MRUnit) — Chris White, Apr 13 '12 at 10:39

Jim Donofrio · Accepted Answer · 2012-04-16T03:59:19.210

You are getting a NPE because the SerializationFactory is not finding an acceptable class implementing Serialization in io.serializations.

MRUnit had several bugs related to serializations besides Writable including MRUNIT-45, MRUNIT-70, MRUNIT-77, MRUNIT-86 at https://issues.apache.org/jira/browse/MRUNIT. These bugs involved the conf not getting passed to the SerializationFactory constructor correctly or the code required a default constructor from the Key or Value which all Writables have. All of these fixes appear in Apache MRUnit 0.9.0-incubating which will get released sometime this week.

Cloudera's 0.20.2-cdh3u2 MRUnit is close to Apache MRUnit 0.5.0-incubating. I think that your code may still be a problem even in 0.9.0-incubating, please email your full code example to mrunit-user@incubator.apache.org and the Apache MRUnit project will be happy to take a look at it

This will compile now MRUNIT-99 relaxes the restriction on K2 type parameter to not have to be Comparable

i switched to 0.9.0-incubating and now it runs mostly. got another exception: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.CharSequence. looking into that right now — Salandur, Jun 27 '12 at 14:30

Use MRUnit and AVRO together

1 Answers1