11

Over the last 6 months our test team have been using selenium webdriver to test our web based products. We have had great success with it and continue to use it on a daily basis. We use visual studio since we are a .net shop to write our c# unit tests. We don't use any other testing frameworks for .net. We have up until recently been running our automation tests manually through the test explorer window in visual studio (2013), checking on the results and then logging them into a spread sheet. I know this isn't ideal but we don't have that many tests so this has suited us fine thus far. However, the number of automation tests we will be required to write and maintain is due to rapidly increase over the next few months.

I have recently been playing around with creating batch files for calling vstest.console.exe and its various commands and then adding those logs to a server. Not ideal. I know I can still do so much more, specifically integrate some sort of CI server.

We are already using team foundation server and have various virtual servers (all running windows 8.1) at our disposal so I thought about taking advantage of this so I began looking into Jenkins. Trouble is, I'm not finding much information regarding Jenkinks and c#. It looks primarily geared to a java setup. Am I missing something? What little information I have found is seriously outdated and didn't work for me.

I got as far as setting it up and installing the vstest.console.exe plugin but couldn't get a simple test to run. A current step by step guide that doesn’t pre-date 2012 would be great :) Do you guys think Jenkins is the way to go for c# and the .net framework? Is there a "standard" used within the c# community? I have heard of cruise control and I’m going to check that out. Is it a viable alternative? Easier to use with .net?

Here is essentially what we need:

  1. Continue writing our tests inside visual studio and creating c# unit tests
  2. Schedule a run of our unit tests on a remote / local server
  3. Write out a result / log file - nice reporting features on fails / passes would be great
  4. Email said file to qa / dev teams

I'm hoping some of you guys have been down this road once and can share some insights

Banjaxx
  • 624
  • 2
  • 18
  • 34
  • 4
    Using CI to run only scheduled tests seems overkill. Using Task Scheduler to invoke a batch file would be simpler. There's also TFS itself. Also the Visual Studio test runner, AFAIK, outputs trx files. Not sure how easy it is going to be to convert one of those into a readable result/log file. I'm interested to see how the answers to that one pan out. – John O. Dec 09 '14 at 19:48
  • 3
    TFS already has this built in. Adding Jenkins on top of all this is adding an horrendous amount of extra complexity and duplication for little to no gain. Look into TFS properly, it can do all of this out of the box (especially as you are using the inbuilt MSTest). – Arran Dec 09 '14 at 22:22
  • 1
    Thanks guys. Will look into TFS in more depth – Banjaxx Dec 10 '14 at 10:32
  • 1
    TFS is good, another (better, according to our build guy) option is [Teamcity](https://www.jetbrains.com/teamcity/). – Nathan Cooper Jan 05 '15 at 19:37

2 Answers2

11

It is possible to use Jenkins to run tests via batch scripts, reporting back to Jenkins via the NUnit or MSTest plugins. To do this, simply call the test runner from a Jenkins-executed script (see links below). The primary reason for doing this in my shop is that Jenkins is used to automate the build process, and automated tests are run every time new code is promoted. If you don't use Jenkins for build automation and reporting - i.e. you just want scheduling - the most basic solution would be Task Scheduler (as John O indicated). Plus, if you are using MSTest rather than NUnit then, as others have suggested, it is better to have a look at TFS.

If you really want to use Jenkins with MSTest, the following links may be useful from a configuration perspective:

Error trying to run mstest on jenkins - 2012

Example of running MSTest from Jenkins from above link:

"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Example.Tests\bin\Debug\Example.Tests.dll /resultsfile:TestResults.trx

Jenkins MSTestRunner plugin is unable to launch mstest.exe - 2014

If you can/want to use NUnit, check out the following:

How do you run NUnit tests from Jenkins?

Execute NUnit with Jenkins

Actually, for reporting purposes, logging to Excel or something similar isn't a horrible idea (particularly if your organization uses Sharepoint). Worst comes to the worst and you can't get anything to work, then one solution would be automating this reporting process by using Excel's COM Object Model to directly write results to the spreadsheet.

Would still suggest that TFS is your best bet, however.

Vivek Parekh
  • 1,075
  • 9
  • 25
Orphid
  • 2,722
  • 2
  • 27
  • 41
0

I am going to give you some information on how the test can be handle from CI not exactly how to configure Jenkins or other CI server. This might be a partial answer to your question but at least something. Keep that in mind selenium tests always need a live site to point to and before you kick off the tests site deployment should happen(either manually or automatic). I prefer running the test suite once at night and let it go for whole night. You can simply accomplish this with the help of NUnit console, batch and windows task scheduler. See my answer here how to do that. Sure enough this can be accomplish from any other CI server. With NUnit Command switch you can export result automatically also.

Community
  • 1
  • 1
Saifur
  • 16,081
  • 6
  • 49
  • 73