14

I've got Visual Studio 2010, and we have two VS solutions we work with. The first is the web application, and the second is strictly for SpecFlow tests. Having two instances of Visual Studio running at the same time just to run SpecFlow features is eating all the available RAM causing things to slow down.

I've done some searching on Google and here on StackOverflow, plus perused the MS documentation on the MSTest command line tool, but I haven't found the answer. The full SpecFlow test suite takes ~45 minutes to complete, and I really only need to run a few scenarios.

I was wondering if there is a way to run individual SpecFlow features, and even individual scenarios, from the command line using MSTest?

Greg Burghardt
  • 17,900
  • 9
  • 49
  • 92
  • You might also consider looking at NCrunch (http://NCrunch.net) or ContinuousTests (http://www.continuoustests.com). Both of these tools are designed to run your tests as quickly as possible. I use NCrunch which has an *only run the tests affected by my change* mode, and would be just what you are after – AlSki Dec 17 '13 at 17:12

5 Answers5

19

Behind the scene specflow tests are just regular mstest unit tests. So you should be able to run them the same way using something like:

To run a specific scenario:

mstest /testcontainer:tests.dll /test:GivenMyScenarioWhenIDoSomeStuff

To run a several specific scenario you can use the /test flag multiple times:

mstest /testcontainer:tests.dll /test:GivenMyScenarioWhenIDoSomeStuff /test:GivenMyScenarioWhenIDoSomemthingElse

To run a feature

mstest /testcontainer:tests.dll /test:MyFeatureName

If you add tags on your scenarios using @MyTag for example, you could also use the option 

/category:MyTag to filter down the scenarios to run.

Please have a look to the generated code behind of your feature files to get a clue of how things actually work, if you are familliar with mstest it should be pretty straightforward.

foobarcode
  • 2,279
  • 1
  • 23
  • 24
  • It looks like this is the answer, however I'm butting my head against another issue. I have installed all the test project's dependencies via NuGet, which throws all those DLLs into the "packages" directory. The `mstest` command line utility expects all the class libraries to exist in the current working directory. I installed the `SpecBind` NuGet package, which does give me the missing DLL file (SpecBind.CodedUI.dll) but it's not getting copied to my build directory. I'll have to do some digging into this unless you've got some suggestions (I'm uses CodedUI tests for this project). – Greg Burghardt Dec 17 '13 at 13:49
  • I'm marking this as the answer, because I'm butting my head against a different issue -- probably one due to the setup of my Visual Studio solution. Thanks! – Greg Burghardt Dec 17 '13 at 13:58
  • 4
    Use Vstest.console.exe instead of mstest. Vstest.console.exe will automatically load all your nuget packages as well as other libraries. E.g Vstest.console.exe mytests.dll – Mohamed Jan 21 '15 at 16:38
  • How do you do an "AND" or "OR" condition on the tags - if you wanna run multiple test case that have different tags? – Monnie_tester Apr 04 '21 at 23:12
8

Now that SpecFlow 3.0 has been released we can use SpecFlow with .NET Core. The CLI tool for .NET Core is dotnet and tests are run like this if you use MSTest (vstest):

dotnet test

If the tests are in a specific project you can specify the project like this

dotnet test TestProject

where TestProject is the name of the project. You can skip the project name if you want to, but specifying it will make dotnet look in only that project. To list all the tests in the project you can use the -t flag:

dotnet test TestProject -t

To run only specific tests you can use the --filter flag:

dotnet test TestProject --filter ShouldBeSuccess_1

where ShouldBeSuccess_1 is the name of the test. The argument after --filter is an expression, and not necessary the name of the test If you had a test called ShouldBeSuccess_12 it would also run. You can see the rules for --filter here.

To only run the tests in a specific category you can use TestCategory:

dotnet test TestProject --filter TestCategory=ci

where ci is the category name. To add a test to a category you use tags.

To create the results file you have to use the --logger flag:

dotnet test TestProject --logger trx

Here it's used to create a trx results file.

Mika Sundland
  • 18,120
  • 16
  • 38
  • 50
2

There is a nuget package named "Specrun.Specflow" download that. And it will change your app.config and set unitTestProvider name="SpecRun", so you can remove unitTestProvider name="MSTest" or "NUnit", now on saving App.config changes, visual studio prompts you to regenerate your feature files, click on Yes and now build a solution, What you will see is your test files have been regenerated. Now in your Command Prompt go to C:\Users\\Documents\Visual Studio 2015\Projects\ and type runtests.cmd , it should trigger all your Feature files directly.

UNG
  • 673
  • 1
  • 6
  • 17
  • 8
    Specrun.Specflow is the commercial product of Specflow and costs 150 dollars per user. This revenue stream appears to be why there is little to no support for running spec flow in any of the standard open source runners. Paid for runners is really unhelpful for CI environments and presents an organisational barrier to usage even for wealthy corporations. This of course is not your fault but I thought it should be mentioned as a major caveat to your suggestion. – Amias Sep 27 '17 at 10:24
  • As above, it's not helpful to deviate away from the original requirement and suggesting a commercial version as a solution. I'm still trying to figure out how to run the Specflow (with Mstest) features from command line – vijay pujar Jan 27 '21 at 18:41
2

With MSTest v2, you can´t use mstest. You can use vstest.console.exe instead.

Example:

vstest.console.exe "Automation.SpecFlow\bin\Release\Automation.SpecFlow.dll"

https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019

If you want to run all scenarios in a single feature file, then add the /trait flag:

vstest.console.exe "Automation.SpecFlow\bin\Release\Automation.SpecFlow.dll" /trait:"My Feature"

And this runs all scenarios in the feature files that begin with:

Feature: My Feature
    In order to ...
    As a ...
    I want to ...

Scenario: 1
    ...

Scenario: 2
    ...
Greg Burghardt
  • 17,900
  • 9
  • 49
  • 92
alansiqueira27
  • 8,129
  • 15
  • 67
  • 111
  • Your answer reminds me. I figured out how to filter it to a single feature file too. I edited your answer to add that info. +1 – Greg Burghardt Jun 04 '19 at 16:07
0

I tried the tags technique but it didn't work, I'm using an older version of SpecFlow. So, I went to the .feature.cs file related to a feature file and searched for TestMethodAttribute()

[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]

I added the TestCategory attribute on top of this, just like the following: 

[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategory("MyCat")]

Build and compile and the command worked like a charm with 

/Category:MyCat

I hope someone will find the answer useful.

Naeem A. Malik
  • 995
  • 4
  • 19
  • You should not edit .feature.cs file because it is regenerated every time you edit the .feature file. You should add a tag to the .feature file (as @someTag) and then the .feature.cs file will contain [Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("someTag")] – Youcef Kelanemer Mar 08 '19 at 20:24
  • 1
    It is possible to create a partial class separately and add attributes to that instead of messing with auto-gen files. – Naeem A. Malik Mar 11 '19 at 13:18