0

Right now my Coded UI Tests use their app.config to determine the domain they execute in, which has a 1-1 relationship with environment. To simplify it:

  • www.test.com
  • www.UAT.com
  • www.prod.com

and in App.config I have something like:

<configuration>
    <appSettings>
        <add key="EnvironmentURLMod" value ="test"/>

and to run the test in a different environment, I manually change the value between runs. For instance the I open the browser like this:

browserWindow.NavigateToUrl(new Uri("http://www."
                + ConfigurationManager.AppSettings.Get("EnvironmentURLMod")
                + ".com"));

Clearly this is inelegant. I suppose I had a vision where we'd drop in a new app.config for each run, but as a spoiler this test will be run in ~10 environments, not 3, and which environments it may run may change.

I know I could decouple these environment URL modifications to yet another XML file, and make the tests access them sequentially in a data-driven scenario. But even this seems like it's not quite what I need, since if one environment fails then the whole test collapses. I've seen Environment Variables as a suggestion, but this would require creating a test agent for each environment, modifying their registries, and running the tests on each of them. If that's what it takes then sure, but it seems like an enormous amount of VM bandwidth to be used for what's a collection of strings.

In an ideal world, I would like to tie these URL mods to something like Test Settings, MTM environments, or builds. I want to execute the suite of tests for each domain and report separately.

In short, what's the best way to parameterize these tests? Is there a way that doesn't involve queuing new builds, or dropping config files? Is Data Driven Testing the answer? Have I structured my solution incorrectly? This seems like it should be such a common scenario, yet my googling doesn't quite get me there.

Any and all help appreciated.

Ryanman
  • 880
  • 1
  • 8
  • 20

1 Answers1

1

The answer here is data driven testing, and unfortunately there's no total silver bullet even if there's a "Better than most" option.

Using any data source lets you iterate through a test in multiple environments (or any other variable you can think of) and essentially return 3 different test results - one for each permutation or data row. However you'll have to update your assertions to show which environment you're currently executing in, as the test results only show "Data Row 0" or something similar by default. If the test passes, you'll get no clue as to what's actually in the data row for the successful run, unless you embed this information in the action log! I'm lucky that my use case does this automatically since I'm just using a URL mod, but other people may need to do that on their own.

To allow on-the-fly changing of what environments we're testing in, we chose to use a TestCase data source. This has a lot of flexibility - potentially more than using a database or XML for instance - but it comes with its own downsides. Like all data driven scenarios, you have to essentially "Hard Code" the test case ID into the decorator above your test method (Because it's considered a property). I was hoping we could drop an app.config into the build drop location when we wanted to change which test case we used, at least, but it looks like instead we're going to have to do a find + replace across a solution instead.

If anyone knows of a better way to decouple the test ID or any other part of the connection string from the code, I'll give you an answer here. For anyone else, you can find more information on MSDN.

Ryanman
  • 880
  • 1
  • 8
  • 20