Configure rspec to show 'E' error when running an erroneous test

Question

TL;DR: How can I stop rspec aborting all specs when it encounters an error?

When running rails' built in testing suite, I get output similar to the following:

...F..F.EE...E

Finished in 0.64396 seconds
14 examples, 2 failures, 3 errors 

The . represents a passing test, the F a failing test and the E an erroneous test. A failing test means the code of your rails application is failing, and an erroneous test means the code of your actual test is failing. Erroneous tests are never good.

What I do like about this though, is that when the program encounters an E, it keeps on going through all the other tests.

I'm using rpsec, and while I do like it, I sort of hate how, when it encounters an erroneous spec, it completely gives up the ghost, and quits all specs, like this:

12:39:37 - INFO - Running: spec
/usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/validations.rb:57:in `save!': Validation failed: Email has already been taken (ActiveRecord::RecordInvalid)
    from /usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/attribute_methods/dirty.rb:41:in `save!'
    from /usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/transactions.rb:275:in `block in save!'
    from /usr/local/rvm/rubies/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/activerecord-4.0.1/lib/active_record/transactions.rb:326:in `block in with_transaction_returning_status'

While I do like the error reporting, I really want it to carry on running all specs when it encounters an erroneous spec. Rather than the above output, I really want something along the lines of the following:

12:40:01 - INFO - Running: spec
...F..F..E

Failures:

  1) User A new user can be created email should == "awesomedog@hotmail.co.uk"
     Failure/Error: its(:email) { should == "awesomedog@hotmail.co.uk" }
       expected: "awesomedog@hotmail.co.uk"
            got: "awesomedozg@hotmail.co.uk" (using ==)
     # ./spec/model/user_spec.rb:11:in `block (3 levels) in <top (required)>'

  2) User A new user can be created should not be valid
     Failure/Error: it { should_not be_valid }
       expected valid? to return false, got true
     # ./spec/model/user_spec.rb:20:in `block (3 levels) in <top (required)>'

Errors:

  1) `save!': Validation failed: Email has already been taken (ActiveRecord::RecordInvalid)

Finished in 0.64396 seconds
10 examples, 2 failures, 1 error

Failed examples:

rspec ./spec/model/user_spec.rb:11 # User A new user can be created email should == "awesomedog@hotmail.co.uk"
rspec ./spec/model/user_spec.rb:20 # User A new user can be created should not be valid

I'm using factory-girl and factories in my specs, and data-base cleaner to clean my database between each spec. I'm using rspec-guard to run all specs on the event of any project file (apart from those in tmp,log or db) being saved. Because rpsec keeps wimping out if it hits an error, I'm getting this error:

How can I clean my database between erroneous rspec specs?

Basically, database-cleaner is configured to clean my database when a spec begins and when a spec finishes. Because rspec quits in the middle of a spec when it hits an error, database-cleaner doesn't detect that the spec has finished, and so never cleans my database. This means I have to manually empty it with my database shell.

I'd also prefer to see the state of all my other specs, even if one is erroneous! Rspec is really silly in this regard methinks!

Here are my factories and my spec:

spec/model/user_spec.rb:

require 'spec_helper'

describe User do
    context "Valid user" do
        user = FactoryGirl.create(:user_with_all_valid)

        subject { user }
        its(:first_name) { should == "Jimmy" }
        its(:last_name) { should == "Thehat" }
        its(:profile_name) { should == "Jimbohatboy893" }
        its(:email) { should == "awesomedog@hotmail.co.uk" }
        its(:password) { should == "thisisasupersecretpassword12234234" }
        its(:password_confirmation) { should == "thisisasupersecretpassword12234234" }
    end
end

spec/factories.rb:

FactoryGirl.define do

    factory :user_with_all_valid, class: User do
        first_name "Jimmy"
        last_name "Thehat"
        profile_name "Jimbohatboy893"   
        email "awesomedog@hotmail.co.uk"
        password "thisisasupersecretpassword12234234"
        password_confirmation "thisisasupersecretpassword12234234"
    end
end

Answer 1

You need to have a unique email each time due to the validation on the model.
btw the other 7 or 8 are passing (dots).

Try adding:

new_user_time = Time.new

to save the current time. Then pass that in to the create user factory. Then use it to check that the users email is correct.

Answer 2

Use sequences as detailed at: https://github.com/thoughtbot/factory_girl/blob/master/GETTING_STARTED.md#sequences

Unique values in a specific format (for example, e-mail addresses) can be generated using sequences. Sequences are defined by calling sequence in a definition block, and values in a sequence are generated by calling FactoryGirl.generate:

# Defines a new sequence
FactoryGirl.define do
  sequence :email do |n|
    "person#{n}@example.com"
  end
end

FactoryGirl.generate :email
# => "person1@example.com"

FactoryGirl.generate :email
# => "person2@example.com"
...

The just have your spec check that part of the email address is correct, e.g. the @something.com or use a regular expression to compare, e.g. match(/person.*@something\.com/)

Answer 3

Your issue is this line:

context "Valid user" do
  user = FactoryGirl.create(:user_with_all_valid)  # Right here!
  ...

Test setup should always be done in a before block; variable definition is done in a let block. Doing it in your test definition will cause rspec to fail as described. In general, you should only ever have rspec directives (before, after, let, subject, it, its, etc) in a context block. If you're running test setup, application code, whatever, you're going to open yourself up to this kind of problem. What you want is:

context "Valid user" do
  let(:user) { FactoryGirl.create(:user_with_all_valid) }

or:

context "Valid user" do
  before { @user = FactoryGirl.create(:user_with_all_valid) }

The suite will then run the whole thing as expected, and report failures for the tests that fail to do their defined setup (the user definition) properly, rather than bombing out of the whole suite.