MapReduce - Mock with Mokito

Question

I have the a reducer class that I wanted to write test cases:

Reduce class:

public class MyReducer extends Reducer<Text, Text, NullWritable, Text> {
    private static final Logger LOG = LogManager.getLogger(MyReducer.class);
    public static List<String> l1 = new ArrayList<String>();
    String id = null;
    private MultipleOutputs<NullWritable, Text> mos;

    @Override
    public void setup(final Context context) throws IOException, InterruptedException {
        mos = new MultipleOutputs<NullWritable, Text>(context);

         final Path[] uris = DistributedCache.getLocalCacheFiles(context.getConfiguration());

        try {
            final BufferedReader readBuffer1 = new BufferedReader(new FileReader(uris[0].toString()));
            String line;
            while ((line = readBuffer1.readLine()) != null) {
                l1.add(line);
            }
            readBuffer1.close();
        } catch (Exception e) {
            LOG.error(e);
        }
    }

    public void reduce(final Text key, final Iterable<Text> values, final Context context)
            throws IOException, InterruptedException {

        final String[] key1 = key.toString().split("-");
        final String keyA = key1[10];
        final String date = key1[1];

/* Some condition check */ 

           mos.write(NullWritable.get(), new Text(inputEventValue), keyA + "//date=" +
                    date.substring(0, 4) + "-" + date.substring(4, 6));

       }

    @Override
    public void cleanup(final Context context) throws IOException, InterruptedException {
        mos.close();
    }

}

Test Case looks like :

@RunWith(MockitoJUnitRunner.class)
public class MyTest {

   @Mock
    private MyReducer.Context mockContext;

    MyReducer reducer;
    MultipleOutputs<NullWritable, Text> mos;

    @Before
    public void setUp() {
        reducer = new MyReducer();
    }


   @Test
    public void myReducerTest() throws Exception {

        MyReducer spy = PowerMockito.spy(new MyReducer());
        doNothing().when(spy).setup(mockContext);

        mos = new MultipleOutputs<NullWritable, Text>(mockContext);
        List<Text> sline = new ArrayList<>() ;
        List<String> l1 = new ArrayList<String>();
        l1.add(“1234”);
        sline.add(new Text(“xyz”));
        Whitebox.setInternalState(MyReducer.class,”l1", l1);
        Whitebox.setInternalState(MyReducer.class,"mos",mos);
        reducer.reduce(new Text(“xyz-20200101-1234),sline,mockContext);

    }

    @After
    public void tearDown() throws Exception {
        /*
         * this will do the clean up part
         */
        verifyNoMoreInteractions(mockContext);
    }

When running in Debug mode it goes to the reducer's reduce method and fails with NullPointerException where mos write statement is?

Complete Stack trace:

java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getNamedOutputsList(MultipleOutputs.java:196)
    at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.<init>(MultipleOutputs.java:324)
    at MyTest.myeducerTest
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
    at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:86)
    at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:94)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:294)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:127)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:282)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:207)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:146)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:120)
    at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
    at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:118)
    at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:101)
    at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
    at org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:53)

Mocking mos errors as mos is not a static.

Any suggestion.

Junit - ReduceDriver, withInput, withOutput,testRun  doesn't work.

Thanks.

I tried mocking Multiple outputs as suggested:

import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;

@Mock private MyReducer.Context mockContext;

    List<String> namedOut = new ArrayList<>();
    namedOut.add("NM1");
    namedOut.add("NM2");

MultipleOutputs spy = PowerMockito.spy(new MultipleOutputs<>(mockContext)); when(spy, "getNamedOutputsList(mockContext)").thenReturn(namedOut);

But this gives me error : org.powermock.reflect.exceptions.MethodNotFoundException: no method found with name 'getNamedOutputsList(() anyObject())' with parameter types : [] in class org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.

Your (incomplete) stacktrace does not really indicate that it fails from the line `mos.write`, so I didn't consider that in my answer. You might want to add the complete stacktrace if it does not fail during the construction of `MultipleOutputs`. — second, Feb 10 '20 at 13:27
`MultipleOutputs.` confirms that it is the constructor. Did you give what I wrote in my answer a try? — second, Feb 10 '20 at 19:34
I still think its `cleaner` to just mock the configuration object. I didn't test that, so in case it does not work for you I added a link for the reflection stuff. — second, Feb 11 '20 at 12:03
I did mocking config but that didn't work. See the code above that I tried mocking/spy on MultiplleOutputs for getNamedOutputs method which is private static. Any suggestion pls — Aavik, Feb 12 '20 at 10:27
You used the wrong syntax for mocking static methods. You have to declare a `mockStatic` for the class and then use the `MultipleOutputs.getNamedOutputsList(...)` inside the `when`. You do not need a `spy` for this part. — second, Feb 12 '20 at 15:25

second · Answer 1 · 2020-02-12T15:22:14.900

Looks like you did not define what mockContext.getContext() should return for your test, so it returns null and fails.

Based on this sourcecode the methods looks like this (so you might use a different version):

private static List<String> getNamedOutputsList(JobContext job) {
   List<String> names = new ArrayList<String>();
   StringTokenizer st = new StringTokenizer(
     job.getConfiguration().get(MULTIPLE_OUTPUTS, ""), " ");
   while (st.hasMoreTokens()) {
     names.add(st.nextToken());
   }
   return names;
}

JobContext seems to refer to your mock Reducer.Context mockContext, so you need to define the appropriate behaviour so that it returns what it is supposed to return.

Note that this call originates from the constructor of MultipleOutputs.

Also take note of the static getCountersEnabled method that is invoked from the constructor and interacts with the context.

Mocking mos errors as mos is not a static.

You could probably use reflections to put a mocked version of mos into your MyReducer class.

Check here for some example on how to mock a private static field.

Edit:

If you try to mock the conig do it like this:

Configuration config = Mockito.mock(Configuration.class);
when(mockContext.getConfiguration()).thenReturn(config);

As far as I see the get that are invoked on the configuration object always provide a default value, so it shouldn't matter if the key/value pair is in there or not.

This is what I have tried so far, but no luck. Suggestion from here: — Aavik, Feb 12 '20 at 09:37
@PrepareForTest({MultipleOutputs.class, MyReducer.class}) List namedOut = new ArrayList<>(); namedOut.add("NM1"); namedOut.add("NM2");MultipleOutputs spy = PowerMockito.spy(new MultipleOutputs<>(mockContext)); PowerMockito.doReturn(namedOut).when(spy, MultipleOutputs.class,"getNamedOutputsList",any(JobContext.class)); — Aavik, Feb 12 '20 at 12:25
This erros with InvokationTargetException. It goes into the MultipleOutputs getNamedOutputsList method — Aavik, Feb 12 '20 at 12:28
What needs to be done to define the behaviour of mockContext ? — Aavik, Feb 12 '20 at 12:29
I have added a simple example on how the config class should be mocked. You might need to search for the correct `Configuration` class. — second, Feb 12 '20 at 15:23

MapReduce - Mock with Mokito

Reduce class:

Test Case looks like :

Complete Stack trace:

1 Answers1