0

When doing Behavior Driven Development, is there a standard way to decide how to make the tradeoff between simple tests and too many tests?

I'm working on implementing BDD for a new project, and I've recently read through the book BDD In Action by John Smart. The book recommends not intermixing When and Then steps. This seems like it would likely result in more tests, which may be desirable from a simplicity perspective but could increase the amount of time tests take to run.

I'm considering a scenario like the following:

Given that a user exists with the following details:
  first name | email
  Bill       | bill@example.com
When I submit the forgot password form
Then bill@example.com should receive a forgot password email
And the email body should contain Bill
When I visit the password reset link
And I use ;v'H~N0et,)S}*VX@fQH9=sm@i1jw|'f as my new password
Then I should be logged in
When I log out
And I log in with password ;v'H~N0et,)S}*VX@fQH9=sm@i1jw|'f
Then I should be logged in

This obviously intermixes multiple When and Then steps, but the alternative is having 3 scenarios:

Given that a user exists with the following details:
  first name | email
  Bill       | bill@example.com
When I submit the forgot password form
Then bill@example.com should receive a forgot password email
And the email body should contain Bill

Given that I have a password reset email
When I visit the password reset link
And I use ;v'H~N0et,)S}*VX@fQH9=sm@i1jw|'f as my new password
Then I should be logged in

Given that I have a password reset email
When I visit the password reset link
And I use ;v'H~N0et,)S}*VX@fQH9=sm@i1jw|'f as my new password
When I log out
And I log in with password ;v'H~N0et,)S}*VX@fQH9=sm@i1jw|'f
Then I should be logged in

Is there a recommended way to implement a test like this? Is it OK to intermix the steps in this type of scenario?

Maybe the solution is to make sure the reset process between scenarios doesn't contribute significantly the testing, but this could be difficult if we need to reset the database between each test.

Joshua Dwire
  • 5,415
  • 5
  • 29
  • 50

2 Answers2

1

The BDD purist would tell you to make them separate scenarios. It does mean a larger number of tests, but the advantage here is each test only has 1 reason to fail.

The BDD pragmatist in me would tell you it is fine to intermingle When and Then steps occasionally. Just don't get in the habit of doing this.

If the feature you are testing changes often, I would go with splitting it up into multiple scenarios with only a single Then in each scenario. If the password reset feature does not change that often, I would probably go with a single bowl of When and Then spaghetti and call it a day. If the test becomes "flaky" and fails often, I would consider splitting it into separate scenarios.

Greg Burghardt
  • 17,900
  • 9
  • 49
  • 92
1

I tend to separate scenarios by capability. You've got a few of them going on there:

  • The capability to ask for, and send, a "reset password" link
  • The capability to reset the password
  • The capability to log in with a new password.

It's true that for the scenarios you've got here, it might be more pragmatic to just combine them. I sometimes do that for trivial cases, especially if there's a thing-that-goes-wrong followed by the same thing-going-right. For instance, I might do this:

Given I'm registered with the password `abcd`
When I try to log in with the password `abcf`
Then it should tell me my credentials are incorrect
When I log in with the password `abcd`
Then it should take me to my homepage.

And yes, strictly speaking, that should be two separate scenarios. They're pretty easy to separate though, so I'm OK with doing this.

But, when you start to run into multiple capabilities, it's usually a sign that the number of scenarios you've got is probably about to explode. You can find the other scenarios by asking, "Is there any other context which, for the same event, produces a different outcome?"

For instance you've got a password reset link.

  • What if the email address bounces?
  • What if the link is more than 14 days old?
  • What if the password provided is the same as the previous password?
  • What if the email address provided in the request isn't registered with your system?

If you combine that scenario, suddenly it's really hard to see what's going on, and which bits of behaviour are being exercised.

In this instance we're using behaviour that's reasonably well-known. It's obvious to us that it's submitting the new password that allows someone to log in. We can tell that we don't just get auto-logged-in by clicking the link.

(Worth noting though that most sites make you log back in with the new password, rather than auto-logging-in after submission of that password, so there's already some interesting behaviour to talk about which is hard if we've got multiple scenarios.)

It's even harder to see what's going on when we start including some of the other scenarios I've listed here.

It gets even harder still when it's a domain we're not familiar with; when conversations around these scenarios are being used to collaborate, to surface misunderstandings, and to pass on knowledge and expertise.

So I'm OK starting with just one scenario, for pragmatism; but be aware that it'll probably need refactoring out into more than one pretty quickly. That's not just in case you need to understand when a scenario fails, but so that you can understand the behaviour of the system when you come to read them.

BDD is more about living documentation than it is about tests. Really great BDD tends to help people change the code without causing bugs, even more than catching them. Single-purpose scenarios can help with that.

This answer I gave on automating whole user journeys (when you deliberately have multiple whens and thens) may give you more insight too.

Lunivore
  • 17,277
  • 4
  • 47
  • 92