I have an integration testing solution. I have my tests described in XML files. In order to capitalize on Visual Studio 2010 testing infrastructure, I have a C# class where every XML test file has an associated method that loads the XML file and executes its content. It looks like this:
[TestClass]
public class SampleTests
{
[TestMethod]
public void Test1()
{
XamlTestManager.ConductTest();
}
[TestMethod]
public void Test2()
{
XamlTestManager.ConductTest();
}
...
[TestMethod]
public void TestN()
{
XamlTestManager.ConductTest();
}
}
Each method name corresponds to an XML file name. Hence, I have to have the following files in my test directory:
- Test1.xml
- Test2.xml
- ...
- TestN.xml
XamlTestManager.ConductTest()
uses the StackTrace class to get the name of the calling method and this way it can find the correct XML test file to load.
I would like to get rid of the extra administration of adding/removing/renaming test methods whenever I change my tests, adding/removing/renaming an XML test file. How can I automagically generate this class or its methods during the compilation process based on the actual XML files in my test directory?
Option 1:
I have considered PostSharp, but it does not allow me to look up the XML files and generate methods on the fly (or was I superficial?).
Option 2:
The other idea was to build a Visual Studio custom tool that generates my code whenever it is executed. The downside here is the deployment. The custom tool needs to be registered to VS. I want a solution that can be committed into a repository, check it out to another computer and use it right away.
(I believe in simplicity. "Check out and run" just simplifies the life of new developers soooooo much, if they do not need to go through a list of thing to install before they can compile run the application.)
Do you have any recommendation, how to get rid of the unnecessary maintenance issue?
EDIT:
For the request of Justin, I add more details. We use Bizunit (fantastic!!!) as the basis of our framework with a truckload of custom made high level test steps. From these steps we can build our test like from lego blocks in a declarative manner. Our steps include like FileDrop, WebService invokation or even polling, firing up a full blown web server to simulate a partner web application, random data generator, data comparing steps etc. Here is an example test xml (in fact XAML):
<TestCase BizUnitVersion="4.0.154.0" Name="StackOverflowSample" xmlns="clr-namespace:BizUnit.Xaml;assembly=BizUnit" xmlns:nib="clr-namespace:MyCompany.IntegrationTest;assembly=BizUnit.MyCustomSteps" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<TestCase.SetupSteps>
<nib:ClearStep FailOnError="True" RunConcurrently="False" />
<nib:LaunchSimulatedApp AppKernelCacheKey="provider" FailOnError="True" FireWakeUpCall="False" PortNumber="4000" RepresentedSystem="MyProviderService" RunConcurrently="False" />
<nib:HttpGetStep FailOnError="True" RunConcurrently="False" Url="http://localhost:10000/Home/StartSvgPolling">
<nib:HttpGetStep.Parameters>
<x:String x:Key="PolledAddress">http://localhost:4000/SvgOutputPort.asmx</x:String>
<x:String x:Key="PollingInterval">10</x:String>
<x:String x:Key="FilterFile"></x:String>
</nib:HttpGetStep.Parameters>
</nib:HttpGetStep>
</TestCase.SetupSteps>
<TestCase.ExecutionSteps>
<nib:DocumentMergeStep FailOnError="True" OutputCacheKey="inputDocument" RunConcurrently="False">
<nib:DocumentMergeStep.InputDocuments>
<nib:RandomLoader BoundingBox="Europe" LinkbackUrlPattern="http://MyProviderService/id={0}" MaxAmount="10" MaxID="100" MinAmount="10" MinID="0" NamePattern="EuropeanObject_{0}" NativeFormat="Svg" RepeatableRandomness="False" UriPrefix="European" />
<nib:RandomLoader BoundingBox="PacificIslands" LinkbackUrlPattern="http://MyProviderService/id={0}" MaxAmount="10" MaxID="100" MinAmount="10" MinID="0" NamePattern="PacificObject_{0}" NativeFormat="Svg" RepeatableRandomness="False" UriPrefix="Pacific" />
</nib:DocumentMergeStep.InputDocuments>
</nib:DocumentMergeStep>
<nib:PushToSimulatedApp AppKernelCacheKey="provider" ContentFormat="Svg" FailOnError="True" RunConcurrently="False">
<nib:PushToSimulatedApp.InputDocument>
<nib:CacheLoader SourceCacheKey="inputDocument" />
</nib:PushToSimulatedApp.InputDocument>
</nib:PushToSimulatedApp>
<nib:GeoFilterStep FailOnError="True" OutputCacheKey="filteredDocument" RunConcurrently="False" SelectionBox="Europe">
<nib:GeoFilterStep.InputDocument>
<nib:CacheLoader SourceCacheKey="inputDocument" />
</nib:GeoFilterStep.InputDocument>
</nib:GeoFilterStep>
<nib:DeepCompareStep DepthOfComparision="ID, Geo_2MeterAccuracy, PropertyBag, LinkbackUrl" FailOnError="True" RunConcurrently="False" Timeout="30000" TolerateAdditionalItems="False">
<nib:DeepCompareStep.ReferenceSource>
<nib:CacheLoader SourceCacheKey="filteredDocument" />
</nib:DeepCompareStep.ReferenceSource>
<nib:DeepCompareStep.InvestigatedSource>
<nib:SvgWebServiceLoader GeoFilter="Europe" NvgServiceUrl="http://localhost:10000/SvgOutputPort.asmx"/>
</nib:DeepCompareStep.InvestigatedSource>
</nib:DeepCompareStep>
</TestCase.ExecutionSteps>
<TestCase.CleanupSteps>
<nib:HttpGetStep FailOnError="True" RunConcurrently="False" Url="http://localhost:10000/Home/StopSvgPolling">
<nib:HttpGetStep.Parameters>
<x:String x:Key="PolledAddress">http://localhost:4000/SvgOutputPort.asmx</x:String>
</nib:HttpGetStep.Parameters>
</nib:HttpGetStep>
<nib:KillSimulatedApp AppKernelCacheKey="provider" FailOnError="True" PortNumber="4000" RunConcurrently="False" />
</TestCase.CleanupSteps>
</TestCase>
This is what it does:
- Invokes a Clear operation on the test subject
- Launches a webserver on port 4000 as a simulated partner app under the name MyProviderService
- Invokes the test subject via HTTP Get to poll the simulated partner
- Creates a new document containing geo info from two random generated content
- Pushes the document to the simulated partner - hence the test subject will pick it up via polling
- The test applies a geo filter on the document
- The deep compare step loads the filtered document as base of comparision, and loads the content of the test subject via a web service
- As clean-up, it stops the polling via an HTTP GET step and kills the simulated partner's web server.
The power of Bizunit is that merges the ease of creating tests in C# with intellisense and ease of maintaining/duplicating it in XAML files. For a quick easy read on how it works: http://kevinsmi.wordpress.com/2011/03/22/bizunit-4-0-overview/