19

For a project I have programmed a wcf service library. It can be hosted in IIS and in a self-hosted service.

For all external systems that are connected, I have provided Mock implementations which give some generic data, so such the service (library) keeps running and doing work. It is a classic automaton / finite-state machine.

While bootstrapping, all data sources are connected. In testing mode, the mock implementations are connected. So when I run tests, the service library is "started" from a self-hosted service, not IIS and the the state machine keeps running and processing data packages.

Is there any way to get some kind of "test coverage" from such a run.

I would really appreciate if I could tell which code paths are hit by the example data I provide from the mock objects. And then provide more testdata to get a higher coverage.

If I could do this without having to provide "lots of extra" testing code, it would be great. I think a lot of cases are already covered from the data provided from the mock objects. But right now I have no starting point for that.

Here are some code examples to give a more clear picture of what is meant. Code is strongly simplified of course.

In a very simple console application to start the service (self hosted version)

static void Main(string[] args)
{
    using (var host = new ServiceHost(typeof(MyServiceLib.Service.MyServiceLib)))
    {
        host.Open();
        Console.ReadLine();
        host.Close();
    }
}

In the service library, a constructor is called from that code

public MyServiceLib()
{
    Task.Factory.StartNew(this.Scaffold);
}

Which does nothing more than starting the state machine

private void Scaffold()
{
    // lots of code deleted for simplicity reasons
    var dataSource = new MockDataSource();

    // inject the mocked datasource
    this.dataManager = new DataManager(dataSource);

    // this runs in its own thread. There are parts that are started on a timer event.
    this.dataManager.Start();
}

public class DataManager : IDataManager
{
     public void Start()
     {
         while (this.IsRunning)
         {
             var data = this.dataSource.getNext();

             if (data != null)
             {
                 // do some work with the data retrieved
                 // lots of code paths will be hit from that
                 this.Process(data);
             }
             else
             {
                 Thread.Sleep(1000);
             }
         }
     }

     public void Process(IData data)
     {
        switch (data.PackageType)
        {
            case EnumPackageType.Single:
            {
                ProcessSingle(data);
                break;
            }
            case EnumPackageType.Multiple:
            {
                ProcessMultiple(data);
                break;
            }
            // here are lots of cases
            default:
            {
                Logger.Error("unknown package type");
                break;
            }
        }
     }
}

What I have tried so far:

  1. OpenCover

with a special test dll that would create the Host as shown above, but the host cannot be created properly, so the testing does not start really. I get a "Host is in fault state" error message. I followed this mini-tutorial. Despite that I get a coverage report with a calculated coverage of about 20%. But the service is just starting, it is not doing any work so far.

  1. Visual Studio Performance Tools

The steps are essentially described in this article. I get a myproject.coverage file, but I cannot view it, because I only have a VS Professional, the coverage seems to be only of use in Test Premium or Ultimate editions.

Besides having tried those two, I will accept any answer showing how to get it up and running with any of those (openCover preferred).

Will accept an answer that shows how to test this setup and get a code coverage while leveraging tools to generate most of the code (as pex would, but after trial I see it does not generate very good code).

Mare Infinitus
  • 8,024
  • 8
  • 64
  • 113
  • http://stackoverflow.com/q/276829 – Robert Harvey Jan 31 '15 at 18:26
  • @RobertHarvey Thank you for the link. I added some clarification to the question. – Mare Infinitus Jan 31 '15 at 18:43
  • 2
    Please don't put "EDIT" in your questions; this isn't a forum. Every post on Stack Overflow already has a [detailed edit history](http://stackoverflow.com/posts/28254987/revisions) that anyone can view. – Robert Harvey Jan 31 '15 at 18:47
  • Um, where's your example? – Robert Harvey Jan 31 '15 at 18:48
  • @RobertHarvey Okay, I just wanted to highlight that the question has changed. What example are you missing? – Mare Infinitus Jan 31 '15 at 18:56
  • The one that you referred to in your question, where you say "by the example data I provide." What example? The general description you have provided is not an example; the general answer to your general description is "find a tool that does what you need." – Robert Harvey Jan 31 '15 at 18:56
  • This is the architecture that is my starting point. I have datasources which I mocked to give data "examples". They keep the statemachine running and cover a lot of code path. Any tool, algorithm or metric can be a solution here. This is not a kind of "please recommend me a tool", but a "What to do in my case" question. – Mare Infinitus Jan 31 '15 at 19:07
  • 2
    Your case does not appear to be unique. Code is code; coverage is coverage. The only way for us to be more specific is to see some of the code that you want to test for coverage. – Robert Harvey Jan 31 '15 at 19:07
  • @RobertHarvey No, this is not a very special case. It is just that I never was in the need to calculate some coverage, which is the case now. Have provided some code example to make clear from what I would love to get some coverage. – Mare Infinitus Jan 31 '15 at 19:27
  • I don't see anything remarkable in the code you've posted. There aren't even any bifurcations (decision trees, `if` statements, `while` loops) that might increase the code's cyclomatic complexity. `this.Process(data);` might have something interesting to test, but you haven't shown us that code. Why are you resisting code analysis tools? – Robert Harvey Jan 31 '15 at 19:32
  • @RobertHarvey The example is kept very simple, just to show what the test is about. There are some loops and some control constructs as added now. I want to know which are hit. And what coverage this is resulting in. I do not see how a code analysis tool will help me here in calculation of a coverage. – Mare Infinitus Jan 31 '15 at 19:41
  • "Code coverage" tools, then, if I'm not being specific enough. That's what they're specifically designed to do: tell you how well your tests cover your code. – Robert Harvey Jan 31 '15 at 19:42
  • Okay, now we have the point of the question. There are no specific tests. At least nothing that leverages nUnit or such, which would quite easily result in a coverage report. There are unit tests, but the do not cover the overall service library with its functionality. This is what the mock objects (in the example there is just one) are about here. I would really appreciate if there is any possibility to calculate a coverage and see which paths are hit. – Mare Infinitus Jan 31 '15 at 19:56
  • @RobertHarvey I feel like I'm missing a very simple yet effective way to get what I need. I'm somehow lost with this. – Mare Infinitus Jan 31 '15 at 20:12
  • @RobertHarvey *Please don't put "EDIT" in your questions* - I am assuming then it is fine to do this in answers though...? – tom redfern Feb 02 '15 at 10:14
  • I'm pretty sure I don't get the point, but let's give it a try... If your processing logic (`MyServiceLib`) lives inside a class library, why don't you simply add a test project that feeds the DataManager with your mocked data source and use OpenCover for the outcome of that test? – MrWombat Feb 03 '15 at 09:24
  • The service in "testing mode" is already scaffolding itself through Dependency injection. Doing that all in tests will give me only a partial coverage with each test. I want an integration of all components. Okay, you can make a test that does all of this, but as said, I did not achieve a way to start the service properly. – Mare Infinitus Feb 03 '15 at 09:58
  • I think you are mixing two things, Unit tests and integration tests. - Unit test use mock data to test each specific function of the code, this is where you get the code coverage. - Integration tests use a test DB and here you just hit all the available exposed functions. In the end if each function works as expected and each function is called correctly everything works. – Pedro.The.Kid Feb 03 '15 at 10:30
  • I do not see any reason to not use mock objects in integration tests. And moreover I do not see any reason to not calculate a code coverage from integration tests. – Mare Infinitus Feb 03 '15 at 16:09
  • @MareInfinitus: I'm pretty sure the *definition* of integration tests is "tests of how well the real components work together"; there does not appear to be anyplace to put the mock components. – SamB Feb 05 '15 at 06:10
  • @SamB I do not know of such a definition. I want to test more than one component before giving up white box testing. This can be seen here: http://en.wikipedia.org/wiki/Integration_testing I do not want Big Bang integration. There are many discussions about that, though. http://programmers.stackexchange.com /questions/205137/mocking-complex-data-operations-in-integration-tests – Mare Infinitus Feb 05 '15 at 06:51
  • @MareInfinitus from your link in wikipedia "Limitations Any conditions not stated in specified integration tests, outside of the confirmation of the execution of design items, will generally not be tested." – Pedro.The.Kid Feb 05 '15 at 13:47
  • @Pedro.The.Kid Exactly. And as stated there, it is blockwise testing of components. If I do not want to test a component, I put a mock for it in the test. It is not easy, but it easier than using the real objects. And it is the only way to have automated integration tests at all. The build system has no access to the other systems involved here. From all I have read about testing, mocking is allowed in integration tests as long as no mocks for system internal instances are used. I just mock external systems. – Mare Infinitus Feb 05 '15 at 17:48
  • Did you try: https://www.jetbrains.com/dotcover/ ? I'm not a big fan of R# test runner but it can be cheaper option than upgrading to VS Premium – Random Feb 09 '15 at 05:52
  • @Random I love R#, but I did not try dotCover so far. – Mare Infinitus Feb 09 '15 at 06:41
  • @MareInfinitus If you are mocking other systems you are not doing a integration test. The point of integration test is to see if the two systems work together not if your system does what you expect, that is a unit test. That's why you have a DEV system for development and then mirror the live system on a test environment for integration tests. And again there is no such thing as code coverage on integration tests. – Pedro.The.Kid Feb 09 '15 at 10:20
  • @Pedro.The.Kid I do not see this. Unit tests test one single module. Integration tests test whether all components of a system work together correctly. If external systems are involved, which is not possible in this case, these are smoke tests or other even higher level tests. There is much discussion on this topic, but the consensus is that integration tests integrate a system and can be automated, which is not the case if external systems are involved here like LDAP and external webservices. – Mare Infinitus Feb 09 '15 at 18:44
  • @Pedro.The.Kid Perhaps you mix up integration tests as described here: http://istqbexamcertification.com/what-is-integration-testing/ with system testing, where a whole integrated system is the unit under test, which is described here: http://en.wikipedia.org/wiki/System_testing – Mare Infinitus Feb 09 '15 at 18:57

4 Answers4

1

It would help to see the operations of the service.

I never tried running such "console kind" application under a coverage tool.

I would suggest writing a test with let's say NUnit (or any other unit testing framework; it's not a unit test, obviously, but the technique fits quite well).

In the test, you open the service host, create a client of the service, let the client execute some operations on your service, and close the service host.

Run this test under a coverage tool, and you should be done.

I've done that with NUnit and NCover about 7 years ago, using their current versions at that time (NCover was free software, if I remember it right).

felix-b
  • 8,178
  • 1
  • 26
  • 36
1

Looks like with OpenCover you are actually getting the coverage, but the service is entering Faulted state, so to you need to catch the faults from your ServiceHost and adress that.

Basically you need some kind of error log, and the first thing i would try is looking in the system event logs (Win+R, eventvwr.msc, Enter).

You can also try to listen to the Faulted events on your ServiceHost:

host.Faulted += new EventHandler(host_faulted);

Here is the link to another SO answer addressing this issue: How to find out the reason of ServiceHost Faulted event

Community
  • 1
  • 1
George Polevoy
  • 7,450
  • 3
  • 36
  • 61
1

I would suggest testing your business logic and not the bootstrap code. I mean testing DataManager class and not the hosting and the initializing code. You can write a unit test, using one of the unit testing frameworks, for example NUnit. Then you can run your tests either in Visual Studio with Resharper Ultimate or in your Continuous Integration with Code Coverage tool, like OpenCover or dotCover to get your code coverage.

[TestFixture]
public class DataManagerTests
{

    [Test]
    public void Process_Single_Processed()
    {
        // Arrange
        IData data = new SingleData();

        DataManager dataManager = new DataManager();

        // Act
        dataManager.Process(data);

        // Assert
        // check data processed correctly

    }
}
Boris Modylevsky
  • 3,029
  • 1
  • 26
  • 42
0

in order to allow your Unit-Test-Framework to determin the coverage you have to host the service within the "runner" of the framework (aka. the process that is executing the tests). The coverage is calculated by and withing the "runner" what means that you can not get coverage if the service is hosted anywhere else. Below I'll add an example how to do this.

Greetings Juy Juka

namespace ConsoleApplication4
{
  using System.ServiceModel; // Don't forgett to add System.ServiceModel as Reference to the Project.

  public class Program
  {
    static void Main(string[] args)
    {
      string arg = ((args != null && args.Length > decimal.Zero ? args[(int)decimal.Zero] : null) ?? string.Empty).ToLower(); // This is only reading the input for the example application, see also end of Main method.
      string randomUrl = "net.tcp://localhost:60" + new System.Random().Next(1, 100) + "/rnd" + new System.Random().Next(); // random URL to allow multiple instances parallel (for example in Unit-Tests). // Better way?
      if (arg.StartsWith("t"))
      {
        // this part could be written as a UnitTest and should be 
        string result = null;
        using (ServiceHost host = new ServiceHost(typeof(MyService)))
        {
          host.AddServiceEndpoint(typeof(IMyService), new NetTcpBinding(), randomUrl);
          host.Open();
          IMyService instance = ChannelFactory<IMyService>.CreateChannel(new NetTcpBinding(), new EndpointAddress(randomUrl), null);
          result = instance.GetIdentity();
          host.Close();
        }
        // Assert.Equals(result,"Juy Juka");
      }
      else if (arg.StartsWith("s"))
      {
        // This part runs the service and provides it to the outside. Just to show that it is a real and working host. (and not only working in a Unit-Test)
        using (ServiceHost host = new ServiceHost(typeof(MyService)))
        {
          host.AddServiceEndpoint(typeof(IMyService), new NetTcpBinding(), randomUrl);
          host.Open();
          System.Console.Out.WriteLine("Service hosted under following URL. Terminate with ENTER.");
          System.Console.Out.WriteLine(randomUrl);
          System.Console.In.ReadLine();
          host.Close();
        }
      }
      else if (arg.StartsWith("c"))
      {
        // This part consumes a service that is run/hosted outoside of the application. Just to show that it is a real and working host. (and not only working in a Unit-Test)
        System.Console.Out.WriteLine("Please enter URL of the Service. Execute GetIdentity with ENTER. Terminate with ENTER.");
        IMyService instance = ChannelFactory<IMyService>.CreateChannel(new NetTcpBinding(), new EndpointAddress(System.Console.In.ReadLine()), null);
        System.Console.Out.WriteLine(instance.GetIdentity());
        System.Console.In.ReadLine();
      }
      else
      {
        // This is only to explain the example application here.
        System.Console.Out.WriteLine("I don't understand? Please use one of the following (Terminate this instance with ENTER):");
        System.Console.Out.WriteLine("t: To host and call the service at once, like in a UnitTest.");
        System.Console.Out.WriteLine("s: To host the servic, waiting for clients.");
        System.Console.Out.WriteLine("c: To contact a hosted service and display it's GetIdenttity result.");
        System.Console.In.ReadLine();
      }
    }
  }

  // Declaration and Implementation of the Service

  [ServiceContract]
  public interface IMyService
  {
    [OperationContract]
    string GetIdentity();
  }

  public class MyService : IMyService
  {
    public string GetIdentity()
    {
      return "Juy Juka";
    }
  }
}
Juy Juka
  • 189
  • 1
  • 9