Stubs / Mocks vs. Service Virtualization..? Yikes

Question

I'm currently investigating unit testing in hard-to-reach areas (that's a very high-level view, I know) and I've come up against this question: stubs / mocks or service virtualization?

I'm reading in pursuit of the answer, but the only resources I can find seem to come from SV vendors (who are obviously biased).

Can anyone think of examples when one is absolutely more appropriate than the other, and why? If the answer is "it depends", then please suggest why / what on. It seems like the same results can be achieved either way - it's a question of time (to develop) or available funds (GreenHat etc. aren't cheap!).

Thanks in advance!

EDIT:

Having checked one of the links(1) posted below, I think this is what I'm getting at:

"Virtual services are just test stubs that you can make yourself.

While you can code your own stubs, once you get past very simplistic behaviors the effort and cost of mocking up all the systems you depend on throughout the software development lifecycle becomes overwhelming. Service Virtualization demands automation in that the simulation and modeling can be conducted by direct observation on the part of software rather than requiring manual coding and adjustment. Otherwise, you may be spending as much time maintaining your stub environments as you do building and testing the application functionality itself."

Same as any tool then basically, eh?

(1) http://servicevirtualization.com/top-10

Answer 1

Great question. Basically, stubs disconnect a test suite from an environment and service virtualization emulates an environment in order to better exercise the true intent of a test.

In more detail...

Stubs provide replacement implementations for objects, methods, or functions in order to remove external dependencies. Stubs are typically used during unit and component testing for two main purposes: 1) To isolate the code under test from the integrated environment 2) To enable testing to proceed when it is not possible to access an external resource or problematic method/function/object.

If you're trying to write a unit test and need to replace a simple call to a database, external libraries (e.g., file I/O) or other system API, stubbing might be perfectly suited for your needs.

While stubs/mocks are typically used to “skip” unavailable system components, service virtualization allows team members to emulate environments (or specific components) and make their behavior available to the entire team. For instance, service virtualization might be used to emulate the behavior of a dependent component (such as a 3rd-party service, database, mainframe, packaged application, etc.) that is evolving, not yet available, or difficult to access / configure for testing.

Service virtualization can represent much more realistic behavior than simple stubs and mocks. If you can access the dependent application, you can capture its current behavior in a "virtual asset" by recording from the live system. Alternatively, you can model virtual assets that represent the anticipated behavior. You can then configure this virtual asset by parameterizing its conditional behavior, performance criteria, and test data. Moreover, you can easily modify the virtual asset to produce the appropriate assortment of fault conditions, exceptions, etc. that should be exercised in order to validate the full range of system behavior.

Parasoft, the company I work for, addresses both of these use cases. Our Development Testing platform facilitates stub generation and management for unit testing, and our Jolt-award winning Parasoft Virtualize product offers service virtualization. See http://www.parasoft.com/service-virtualization and http://www.parasoft.com/development-testing for details.

Answer 2

IMO, Unit tests should not depend on external services, including Service Virtualization.

You would only need to wire those dependencies on external services when writing Integration Tests, Functional Tests, End-to-End Tests.

Though all of those type can be written or driven in test frameworks such as JUnit, they are different type of tests.

Answer 3

Here is an InfoQ article that outlines the differences: Stubbing, Mocking and Service Virtualization Differences for Test and Development Teams

From the article:

Stub: a minimal implementation of an interface that normally returns hardcoded data that is tightly coupled to the test suite. It is most useful when the suite of tests is simple and keeping the hardcoded data in the stub is not an issue. Some stubs are handwritten; some can be generated by tools. A stub is normally written by a developer for personal use. It can be shared with testers, but wider sharing is typically limited by interoperability issues related to software platform and deployment infrastructure dependencies that were hardcoded. A common practice is when a stub works in-process directly with classes, methods, and functions for unit, module, and acceptance testing. Some developers will say that a stub can also be primed, but you cannot verify an invocation on a stub. Stubs can also be communicating "over the wire", for example HTTP, but some would argue that they should be called virtual services in that case.

Mock: a programmable interface observer, that verifies outputs against expectations defined by the test. It is frequently created using a third party library, for example in Java that is Mockito, JMock or WireMock. It is most useful when you have a large suite of tests and a stub will not be sufficient because each test needs a different data set up and maintaining them in a stub would be costly. The mock lets us keep the data set-up in the test. A mock is normally written by a developer for personal use but it can be shared with testers. However, wider sharing is typically limited by interoperability issues related to software platform and deployment infrastructure dependencies that were hardcoded. They most often work in-process directly with classes, methods, and functions for unit, module, and acceptance testing. Mock provides responses based on a given request satisfying predefined criteria (also called request or parameter matching). A mock also focuses on interactions rather than state so mocks are usually stateful. For example, you can verify how many times a given method was called or the order of calls made to a given object.

Virtual service: a test double often provided as a Software-as-a-Service (SaaS), is always called remotely, and is never working in-process directly with methods or functions. A virtual service is often created by recording traffic using one of the service virtualization platforms instead of building the interaction pattern from scratch based on interface or API documentation. A virtual service can be used to establish a common ground for teams to communicate and facilitate artifact sharing with other development teams as well as testing teams. A virtual service is called remotely (over HTTP, TCP, etc.) normally supports multiple protocols (e.g. HTTP, MQ, TCP, etc.), while a stub or mock normally supports only one. Sometimes virtual services will require users to authorize, especially when deployed in environments with enterprise-wide visibility. Service virtualization tools used to create virtual services will most often have user interfaces that allow less tech-savvy software testers to hit the ground running, before diving into the details of how specific protocols work. They are sometimes backed by a database. They can also simulate non-functional characteristics of systems such as response times or slow connections. You can sometimes find virtual services that provide a set of stubbed responses for given request criteria, and pass every other request to a live backend system (partial stubbing). Similar to mocks, virtual services can have quite complex request matchers, that allow to have one response returned for many different types of requests. Sometimes, virtual services simulate system behaviors by constructing parts of the response based on request attributes and data.

It is often difficult to definitely say which of the following categories a test double fits into. They should be treated as spectrums rather than strict definitions.

Answer 4

We meant that SV can represent behavior more realistically than stubs can. Service virtualization do not need to operate as silos; they can represent composite behavior. For example, a call to transfer funds in one virtual endpoint can trigger an account balance update at another. This allows virtual assets to behave in a stateful manner, and allows you to easily model the behavior of an entire system—even if the behavior crosses multiple connections, protocols or interfaces.