0

I am writing a Rails 3.1 app, and I have a set of three cucumber feature files. When run individually, as with:

cucumber features/quota.feature
-- or --
cucumber features/quota.feature:67  # specifying the specific individual test

...each feature file runs fine. However, when all run together, as with:

cucumber

...one of the tests fails. It's odd because only one test fails; all the other tests in the feature pass (and many of them do similar things). It doesn't seem to matter where in the feature file I place this test; it fails if it's the first test or way down there somewhere.

I don't think it can be the test itself, because it passes when run individually or even when the whole feature file is run individually. It seems like it must be some effect related to running the different feature files together. Any ideas what might be going on?

door_number_three
  • 5,843
  • 2
  • 16
  • 21

2 Answers2

4

It looks like there is a coupling between your scenarios. Your failing scenario assumes that system is in some state. When scenarios run individually system is in this state and so scenario passes. But when you run all scenarios, scenarios that ran previously change this state and so it fails.

You should solve it by making your scenarios completely independent. Work of any scenario shouldn't influence results of other scenarios. It's highly encouraged in Cucumber Book and Specification by Example.

Andrei Botalov
  • 20,686
  • 11
  • 89
  • 123
  • Hi Andrey, it would have to be an effect of coupling between feature files, not scenarios, since all the scenarios in each feature file work when the feature files are run individually. That is something I've never heard of. – door_number_three Jul 08 '12 at 23:04
  • @notthequestionyourelookingfor Maybe you have coupling between scenarios in different feature files. – Andrei Botalov Jul 08 '12 at 23:19
  • 1
    This coupling may be due to data not be reset correctly after one of your scenarios in another features. Most likely, the reason it won't even run at the beginning is because one of the other scenarios is leaving data in a bad state. Alternately, you may want to check to make sure you're not using command-line flags when running separately that you're not using when you just call cucumber. – Doug Noel Jul 10 '12 at 13:55
  • 1
    Good call, this was correct. I isolated the problem feature file to just the one feature, then started removing scenarios from the feature file run previously until the test passed. This pointed me to a place where I was setting a server variable in a test (and not properly resetting it as background for other tests). I am surprised this only bit me in the one instance. – door_number_three Jul 11 '12 at 03:13
0

I had a similar problem and it took me a long time to figure out the root cause. I was using @selenium tags to test JQuery scripts on a selenium client. My page had an ajax call that was sending a POST request. I had a bug in the javascript and the post request was failing. (The feature wasn't complete and I hadn't yet written steps to verify the result of the ajax call.) This error was recorded in Capybara.current_session.server.error. When the following non-selenium feature was executed a Before hook within Capybara called Capybara.reset_sessions! This then called 

def reset!
  driver.reset! if @touched
  @touched = false
  raise @server.error if @server and @server.error
ensure
  @server.reset_error! if @server
end

@server.error was not nil for each scenario in the following feature(s) and Cucumber reported each step as skipped. The solution in my case was to fix the ajax call. So Andrey Botalov and Doug Noel were right. I had carry over from an earlier feature. I had to keep debugging until I found the exception that was being raised and investigate what was generating it. I hope this helps someone else that didn't realise they had carry over from an earlier feature.

ccurtis
  • 35
  • 1
  • 3