My partner and I have a suite of tests running nightly on a build server for our project. We use CruiseControl.NET to run the server, and the tests are written using WatiN and NUnit. We have CruiseControl.NET running as a service with access to interact with the desktop on a local system account. Every few times that we run a build, certain tests will fail with error messages such as the following:
Test: cfarmweb.tests.Views.GeneralRegressionTest.DuplicateUsernameTest
Type: Failure
Message: SetUp : System.Runtime.InteropServices.COMException : Creating an instance of the COM component with CLSID {0002DF01-0000-0000-C000-000000000046} from the IClassFactory failed due to the following error: 800704a6. TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
at WatiN.Core.IE.CreateNewIEAndGoToUri(Uri uri, IDialogHandler logonDialogHandler, Boolean createInNewProcess)
at WatiN.Core.IE..ctor(String url)
at cfarmweb.tests.Navigator.SiteNavigator..ctor(String browserName, Boolean visible) in c:\ccworkdir\CFarm\builddir\cfarmweb.tests\Navigator\SiteNavigator.cs:line 35
at cfarmweb.tests.Views.GeneralRegressionTest.MakeNavigator() in c:\ccworkdir\CFarm\builddir\cfarmweb.tests\Views\GeneralRegressionTest.cs:line 34
--TearDown
at WatiN.Core.Browser.OnGetNativeDocument()
at WatiN.Core.DomContainer.get_NativeDocument()
at WatiN.Core.Document.ContainsText(String text)
at cfarmweb.tests.Navigator.SiteNavigator.HasText(String target) in c:\ccworkdir\CFarm\builddir\cfarmweb.tests\Navigator\SiteNavigator.cs:line 213
at cfarmweb.tests.Navigator.SiteNavigator.SignOut() in c:\ccworkdir\CFarm\builddir\cfarmweb.tests\Navigator\SiteNavigator.cs:line 110
at cfarmweb.tests.Views.GeneralRegressionTest.DisposeNavigator() in c:\ccworkdir\CFarm\builddir\cfarmweb.tests\Views\GeneralRegressionTest.cs:line 123
The success of the builds does not seem to be dependent on changes to the code itself, as we have had builds break or be fixed after changes to parts of the program that are unrelated to the tests.
We are both new to the field of software testing (and development in general), but nothing we've found online about this error seems to pertain to our situation. We've seen everything from a system reboot pending to compatibility issues with Internet Explorer 8 to JavaScript errors, but nothing we've tried has fixed the issues. One of the most difficult parts is that it's not consistently reproducible. How can we fix this problem?