4

Using Specflow, I'm writing a suite of scenarios that model monthly payrolls, validating the calculated payments for each month and finally the yearend figures.

The result of each month is cumulative, so each subsequent scenario is dependent on the previous month’s additions and deductions. The payment calculations are written to a database via a third party tool, so creating and destroying test data between scenarios is expensive.

From my experience with testing, I know it's not always possible to ensure the execution sequence of the tests. I can control the sequence of execution with some scenario naming conventions, but can't guarantee that a remote test runner is going to run tests in alphabetical order.

Options I've considered:

  • Run the entire year through a single scenario including lots of given, when, then assertions. This results in a single huge scenario that is difficult to read.
  • Create a concatenation "Given" for each scenario. "Given: All payments to month X have been made". This creates lots of database traffic as each scenario will need to create and destroy test data.

Is there a better way to store state between scenarios and ensure scenarios are executed in the desired sequence?

Cr1spy
  • 241
  • 1
  • 5
  • You may also find this answer helpful, if you map it on your problem and you choos a higher level of abstraction for the type of tests you want to do: http://stackoverflow.com/a/23375756/936469 – realtime Apr 29 '14 at 21:48

1 Answers1

5

Relying on the order of execution of the scenarios is an anti-pattern and should be avoided. Test runners usually don't provide any mechanism to control the order of execution for the same reason. It would be also against the concept of executable specification: the scenario should be understandable (and executable) on its own.

In your case the Given part should prepare the data for the calculation in question, the When should calculate and the Then should check the result of that single calculation.

To reduce the execution time maybe you could try to select "important" scenarios in a way that they test different aspects. Probably it is not necessary to test each month 1-11. You could have one test for the first monthly payroll, one for the 2th month, one to close an entire year, one to start a new year, etc.

It is also a common technique that the Given must not necessarily be done the same way as the "real application" would do (from scratch). Sometimes you can do shortcuts in the test to ensure the prerequisites in a faster and easier way. E.g. you could specify the sums of the previous month if that's all your scenario needs to calculate the next month (instead of letting the app calculate everything from scratch). Of course you have to know what you are doing and you have to consider the risk involved with faking some aspects of the application.

Tz_
  • 2,949
  • 18
  • 13
  • Thanks for the feedback. I've deceided to go with the "Entire year in one" scenario. Providing the previous monthly payment calculations isn't tenable, the calculation is too tricky to mockup. – Cr1spy Aug 25 '11 at 08:11
  • 1
    The "calculation" is too tricky to mock up? Surely the previous month's results can be static data - exactly the data you are trying to store between steps, in fact? – perfectionist Jan 28 '12 at 08:26
  • I know that relying on test order is an anti-pattern, but is that true for all types of testing. For Unit testing, I agree because if you rely on another test, you are bleeding out of the unit. For an acceptance test, I think this rule might become a bit more grey, and I can think of many cases where defining the order in which these tests are run can be very useful. If one test requires data to be captured, and another test checks the capturing of the data, at least to me, I would like to be able to just use the data captured by the previous test. This is my opinion though. Prolly wrong – JonathanPeel Jan 19 '22 at 09:44
  • Although, thinking of how Specflow works, sharing a `When` does not add to much work. – JonathanPeel Jan 19 '22 at 09:46