58

One way of thinking about this is: if we care about the design of the code then EasyMock is the better choice as it gives feedback to you by its concept of expectations.

If we care about the maintainability of tests (easier to read, write and having less brittle tests which are not affected much by change), then Mockito seems a better choice.

My questions are:

  • If you have used EasyMock in large scale projects, do you find that your tests are harder to maintain?
  • What are the limitations of Mockito (other than endo testing)?
palacsint
  • 28,416
  • 10
  • 82
  • 109
RAbraham
  • 5,956
  • 8
  • 45
  • 80
  • 1
    This is a no brainer for me. Mockito any day. EasyMock (with this weird replay() nonsense) is extremely unintuitive. Working with easy mock is like putting a pitch fork in your eye. – Arunav Sanyal Nov 17 '20 at 19:58

5 Answers5

124

I won't argue about test readability, size or testing techniques of these frameworks, I believe they are equal, but on a simple example I'll show you the difference.

Given: We have a class which is responsible for storing something somewhere:

public class Service {

    public static final String PATH = "path";
    public static final String NAME = "name";
    public static final String CONTENT = "content";
    private FileDao dao;

    public void doSomething() {
        dao.store(PATH, NAME, IOUtils.toInputStream(CONTENT));
    }

    public void setDao(FileDao dao) {
        this.dao = dao;
    }
}

and we want to test it:

Mockito:

public class ServiceMockitoTest {

    private Service service;

    @Mock
    private FileDao dao;

    @Before
    public void setUp() {
        MockitoAnnotations.initMocks(this);
        service = new Service();
        service.setDao(dao);
    }

    @Test
    public void testDoSomething() throws Exception {
        // given
        // when
        service.doSomething();
        // then
        ArgumentCaptor<InputStream> captor = ArgumentCaptor.forClass(InputStream.class);
        Mockito.verify(dao, times(1)).store(eq(Service.PATH), eq(Service.NAME), captor.capture());
        assertThat(Service.CONTENT, is(IOUtils.toString(captor.getValue())));
    }
}

EasyMock:

public class ServiceEasyMockTest {
    private Service service;
    private FileDao dao;

    @Before
    public void setUp() {
        dao = EasyMock.createNiceMock(FileDao.class);
        service = new Service();
        service.setDao(dao);
    }

    @Test
    public void testDoSomething() throws Exception {
        // given
        Capture<InputStream> captured = new Capture<InputStream>();
        dao.store(eq(Service.PATH), eq(Service.NAME), capture(captured));
        replay(dao);
        // when
        service.doSomething();
        // then
        assertThat(Service.CONTENT, is(IOUtils.toString(captured.getValue())));
        verify(dao);
    }
}

As you can see both test are fairly the same and both of them are passing. Now, let’s imagine that somebody else changed Service implementation and trying to run tests.

New Service implementation:

dao.store(PATH + separator, NAME, IOUtils.toInputStream(CONTENT));

separator was added at the end of PATH constant

How the tests results will look like right now ? First of all both tests will fail, but with different error messages:

EasyMock:

java.lang.AssertionError: Nothing captured yet
    at org.easymock.Capture.getValue(Capture.java:78)
    at ServiceEasyMockTest.testDoSomething(ServiceEasyMockTest.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Mockito:

Argument(s) are different! Wanted:
dao.store(
    "path",
    "name",
    <Capturing argument>
);
-> at ServiceMockitoTest.testDoSomething(ServiceMockitoTest.java:34)
Actual invocation has different arguments:
dao.store(
    "path\",
    "name",
    java.io.ByteArrayInputStream@1c99159
);
-> at Service.doSomething(Service.java:13)

What happened in EasyMock test, why result wasn't captured ? Is store method wasn't executed, but wait a minute, it was, why EasyMock lies to us?

It's because EasyMock mixing two responsibilities in a single line - stubbing and verification. That's why when something is wrong it's hard to understand which part is causing failure.

Of course you can tell me - just change the test and move verify before assertion. Wow, are you serious, developers should keep in mind some magic order inforced by mocking framework?

By the way, it won’t help:

java.lang.AssertionError: 
  Expectation failure on verify:
    store("path", "name", capture(Nothing captured yet)): expected: 1, actual: 0
    at org.easymock.internal.MocksControl.verify(MocksControl.java:111)
    at org.easymock.classextension.EasyMock.verify(EasyMock.java:211)

Still, it is saying to me that method was not executed, but it was, only with another parameters.

Why Mockito is better ? This framework doesn't mix two responsibilities in a single place and when your tests will fail, you will easily understand why.

Rrr
  • 1,747
  • 3
  • 17
  • 22
  • 1
    I'm sold on trying mockito first. Test maintainability is messy enough withhout mock frameworks getting in the way. – Gary Oct 26 '12 at 03:10
  • I realise this is old... but, it's you mixing the verify and stubbing processes here not EasyMock. It's bad practice to make assertions before verification- thus you can actually remove the 'assertThat' call because the verify will pick it up for you: "FileStore.dao("path", "name", capture(Nothing Captured yet)): expected 1, actual: 0". I will concede the Mockito message is clearer here but that's not enough in this far-too-specific example for it to influence someone's decision about EasyMock! – Bob Flannigon Dec 16 '15 at 12:34
  • 1
    I'm not sure how I mixed something here for EasyMock, this is the way how it works, so if you have a good counter example I would be glad to see it. I think it's a matter of personal opinion wether or not this example is good enough to influence decision on these frameworks and I'm happy that many people found it valuable. – Rrr Dec 16 '15 at 22:38
  • 2
    There's no reason to use nice mocks in this example. You should use a normal mock (i.e. `EasyMock.createMock(FileDao.class)`) and then order of verify doesn't matter and the exception is very clear: `Unexpected method call store("path\", "name", java.io.ByteArrayInputStream@58651fd0): store("path", "name", capture(Nothing captured yet)): expected: 1, actual: 0` – Ian Jones Apr 12 '16 at 10:47
  • 2
    Also EasyMock now has similar @Mock annotations to automate the creation of mocks and inject them into the class under test. – Ian Jones Apr 14 '16 at 09:19
  • I would be mad to see non understandable error message of EasyMock `java.lang.AssertionError: Nothing captured yet`. This error message alone is enough to tell that EasyMock was not fine tuned for developer experience. – HenryNguyen Apr 19 '19 at 09:08
52

if we care about the Design of the code then Easymock is the better choice as it gives feedback to you by its concept of expectations

Interesting. I found that 'concept of expectations' makes many devs put more & more expectations in the tests only to satisfy UnexpectedMethodCall problem. How does it influence the design?

The test should not break when you change code. The test should break when the feature stops working. If one likes the tests to break when any code change happens I suggest to write a test that asserts the md5 checksum of the java file :)

Szczepan
  • 521
  • 4
  • 2
32

I'm an EasyMock developer so a bit partial but of course I've used EasyMock on large scale projects.

My opinion is that EasyMock tests will indeed breaks once in a while. EasyMock forces you to do a complete recording of what you expect. This requires some discipline. You should really record what is expected not what the tested method currently needs. For instance, if it doesn't matter how many time a method is called on a mock, don't be afraid of using andStubReturn. Also, if you don't care about a parameter, use anyObject() and so on. Thinking in TDD can help on that.

My analyze is that EasyMock tests will break more often but Mockito ones won't when you would want them to. I prefer my tests to break. At least I'm aware of what was the impacts of my development. This is of course, my personal point of view.

Ajinkya
  • 22,324
  • 33
  • 110
  • 161
Henri
  • 360
  • 3
  • 2
  • 1
    Yes, I have been thinking the same: with Mockito (and Unitils Mock, a similar mocking API) it's much easier to write tests that continue to happily pass when they shouldn't. I suspect this may be the main reason why Mockito-like APIs, which facilite the creation of excessively "loose" tests, are often considered "easier". – Rogério Jun 08 '10 at 18:06
  • 6
    I'd be interested to see an example that contrasts these 2 approaches... – Armand Jan 28 '11 at 15:23
  • I remember to have copy-pasted tested code to the unit test and "reformatted" it to mocking definitions. Just because it was quicker than writing it off manually. If you start writing tests like this, there is something really wrong. How can a copy of the tested code ever be a good unit test? – Stefan Steinegger Feb 13 '19 at 09:27
  • Another problem with the strict approach is that you need to setup every mock in every unit test (method) from scratch. While a "nice" mock can be largely set up once in a setup method (kind of 'when getName would be called, "Hugo"' would be returend). Each test method only has to write what it makes different to the other test methods. So in average, you have one additional line of code for mocking, then run, then verify whatever you want to have verified. That's actually the biggest difference I ever had when writing unit tests with mocks. 3-liners unit tests. You'll test more cases. – Stefan Steinegger Feb 13 '19 at 09:34
7

I don't think you should be too concerned about this. Both Easymock and Mockito can be configured to be 'strict' or 'nice' the only difference is that by default Easymock is strict wheras Mockito is nice.

As with all testing there's no hard and fast rule, you need to balance test confidence against maintainability. I typically find there are certain functional or technical areas that demand a high level of confidence for which I would use 'strict' mocks. For example we probably wouldn't want the debitAccount() method to be called more than once! However there are other cases in which the mock is really little more than a stub so we can test the real 'meat' of the code.

In the early days of Mockito's life API compatibility was a problem but more tools now support the framework. Powermock (a personal favorite) now has a mockito extension

5

I prefer mockito to be honest. been using EasyMock with unitils and the combination of both oftenly results in exceptions like IllegalArgumentException: not an interface as well as MissingBehaviorExceptions. In both cases though the code and test code are perfectly fine. It appeared that the MissingBehaviorException was due to the fact that mocked objects created with createMock (using classextentions!!) did produce this error. When using @Mock it did work! I do not like that kind of misleading behavior and for me that is a clear indication the developers of it do not know what they are doing. A good framework should always be easy to use and not ambiguous. The IllegalArgumentException was also due to some mingle of EasyMock internals. Also, the recording is not what I want to do. I want to test if my code throws exceptions or not and that it returns the expected results. That in combination with code coverage is the right tool for me. I do not want my tests to break whenever I put 1 line of code above or below the previous one because that improves performance or so. With mockito it is no problem. With EasyMock, it will result tests to fail even though the code is not broken. That is bad. It costs time, thus money. You want to test for expected behavior. Do you really care about the order of things? I suppose in rare occasions you might. Use Easymock then. In other case, I think you'll spend considerably less time using mockito to write your tests.

Kind regards Lawrence

Lawrence
  • 1,035
  • 13
  • 8