14

I've just (today) tried SpecFlow for the first time. I'm playing about by creating a new class library in VS2010 Pro and adding a SpecFlow Feature Definition file.

Thing is, the integration doesn't appear to be working properly, with a variety of different errors. I've selected MsTest as the test runner, because I can't be bothered with invoking NUnit (I'd like to use NUnit in the long term but at the moment I just want to get some BDD code working). The generated code files however continue to reference NUnit - which is obviously wrong, since I've just told SpecFlow to run using MsTest. I've done everything I can think of to invoke the code generation again, including creating a brand new class library project with the MsTest option selected in Tools > Options > SpecFlow.

If I leave the test runner field set to 'Auto' and right-click a feature file, then select 'Run SpecFlow Scenarios' I get an error message "Could not find matching test runner".

If I instead change the test runner field to MsTest, I get a different error message on doing the same thing - "Object Reference not set to an instance of an object". I'm not surprised at this one since it's still trying to run NUnit tests even though I've explicitly asked for MsTest, though obviously it shouldn't nullref and present that to the user.

What am I doing wrong? The documentation is not helpful, and as far as I can see, there's no FAQ.

edit #1: I've established that the actual setting I'm looking for is provided using App.Config using the field <unitTestProvider name="MsTest" />. I can see what's happened - the field in the Visual Studio options menu doesn't seem to modify the project you're currently working on. Thing is, this makes it look like that field doesn't do anything at all. I've now persuaded SpecFlow to generate MsTest classes and run using the MSTest runner.

So now the question morphs into a slightly different one: What (if anything) does the Tools > Options > SpecFlow > Test Runner Tool field do?

Raedwald
  • 46,613
  • 43
  • 151
  • 237
Tom W
  • 5,108
  • 4
  • 30
  • 52

5 Answers5

9

With VS2010 the correct value is MsTest.2010 not MsTest as documented. Change your app.config (for the test assembly) and it will work fine (at least with SpecFlow 1.8)

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  </configSections>
  <specFlow>
      <unitTestProvider name="MsTest.2010" />
      <!-- For additional details on SpecFlow configuration options see https://github.com/techtalk/SpecFlow/wiki/Configuration -->
  </specFlow>
</configuration>
Pete Stensønes
  • 5,595
  • 2
  • 39
  • 62
7

In answer to your latest Question. What is the setting "Tools > Options > SpecFlow > Test Runner Tool" this setting controls what will actually run the tests, not what will generate the test code. If it is set to auto i believe it will look at the App.config file where you have set the unitTestProvider to determine what the best tool is to run the tests. An alternaive Test runner made by the same guys as SpecFlow is SpecRun http://www.specrun.com/

So when you go to run the tests it will use this option. As you have discovered though the code generator uses the config file to determine what type of test it should generate (mstest/nunit..)

If you ran the specfow installer ( https://github.com/downloads/techtalk/SpecFlow/SpecFlowSetup_v1.8.1.msi ) to install all the Visual Studio Intergration components when you change the App.config file it normally promps to regenerate the features using the new provider. The manual way to do this though is to right click the Feature and select "Run Custom Tool"

In regards to documentation have you found the git hub wiki? https://github.com/techtalk/SpecFlow/wiki/Documentation

Ryan Burnham
  • 2,619
  • 3
  • 27
  • 43
3

The way I've read this is that the test runner is entirely different to that of the code generator although that doesn't always make sense when the MsTest runner doesn't know about NUnit (I think). Out of the box, the latest version (v2.3.2) even when installed with SpecFlow.MsTest nuget package (of the same version) does not configure your machine to generate MsTest based classes in the background. I am running VS2017 and have Resharper installed as my 'test runner' but the main requirement for generating MsTest based code is a change to the app.config. As per the wiki documentation you also need the following in your app.config. When you save the config you should be prompted for the files to be regenerated.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="specFlow"
             type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow"/>
  </configSections>
  <specFlow>
    <unitTestProvider name="MsTest" />
  </specFlow>
</configuration>
The Senator
  • 5,181
  • 2
  • 34
  • 49
  • Thanks, this just saved me. I've been stuffing around for ages trying to work this one out. Who the heck though designs a "Design Time" extension that reads a file intended for "Runtime" settings? (Rhetorical question) There should be a corresponding "Unit Test Generation Provider" setting or something along those lines listing installed unit test providers in a drop down list. It's such bad design. – Stephen York May 02 '19 at 06:01
2

We are using ReSharper as a runner for SpecFlow acceptance tests; it worked well right out of the box. Although ReSharper is not free, but it worth every penny...

Void Ray
  • 9,849
  • 4
  • 33
  • 53
1

I was never able to get SpecFlow working right from Visual Studio, I spent some time working on it but never go anywhere. Though I found these instructions on setting up NUnit in Visual Studio 2010 and I use this shortcut to run my SpecFlow tests with good effect.

Overall we use PowerShell to run a lot of tests and I was able to incorporate the NUnit command line runner and SpecFlow report generator into a single script I can run easily.

MichaelF
  • 487
  • 2
  • 12
  • 31
  • Great tip, thanks! I set up the menu button immediately because it looked so useful, so regardless this line of inquiry has provided a handy side-benefit. Shame about SpecFlow though, looks like I'm not the only one to have had issues with it doing what it's supposed to do. The actual product works, but the VS integration isn't quite right. – Tom W Nov 09 '11 at 19:09
  • Yeah I wish it was more integrated but I am ok with that for now. It's use does outweigh that benefit for me right now. Glad the tip helps! – MichaelF Nov 10 '11 at 14:22
  • Have you run the specflow installer or did you add it to your project using NuGet? The installer will install all the integration components like the Visual Studio integration where the nuget install only adds the necessary references. – Ryan Burnham Jan 18 '12 at 00:39
  • I used the SpecFlow installer, I didn't really want NuGet on my machine. Although now I have been using it like this for so long I am more used to it do it doesn't impede me. – MichaelF Jan 18 '12 at 14:56