0

Let's say that I want to run so called web acceptance tests on a modern (as of the day of the question) machine, which, let's say, routinely has somewhere between 16 and 128 logical cores. The number could be different in each particular case, but let's stick with this range for now.

By web acceptance test I mean a test, which opens a web page in one of the browsers (Chrome / FireFox / Edge / ... ) using a driver (e.g. chromedriver / geckodriver, etc...), manipulates a web page in whatever way the test wants, and then "collects" some output (e.g. - does the web page has this or that element OR did it navigate to this or that expected page). The actual detail are irrelevant.

Given that such tests naturally spend most of the time waiting (so that to be sure that once they want to manipulate some web page [element] then it has been loaded for sure) it is then seems reasonable to assume that if I have N logical cores on the machine, then I should be able to spawn at least N of such web acceptance tests.

A typical C# code to do that can be summarized as follows:

namespace WebAcceptanceTests
{
    public static class Chrome
    {
        public static async Task Run(
            Uri uri, 
            Func<ChromeDriver, Task> manipulate, 
            Action<ChromeDriver> validate)
        {
            var chromeDriverService = ChromeDriverService.CreateDefaultService();
            chromeDriverService.HideCommandPromptWindow = true;
            var options = new ChromeOptions();

            // To make Chrome window invisible.
            options.AddArgument("--headless");
            using var driver = new ChromeDriver(chromeDriverService, options);

            try
            {
                driver.Manage().Window.Maximize();
                driver.Navigate().GoToUrl(uri);

                await manipulate(driver);
                validate(driver);
            }
            finally
            {
                driver.Quit();
            }
        }
    }
}

where manipulate performs some "manipulation" of the page (e.g. attempts to click some buttons / enters some text / etc...) and validate performs some validation (e.g. if manipulate entered user name and password and then clicked login button, then did the site actually transitioned to logged in page). The actual details of what these manipulate and validate do are irrelevant. However, manipulate is a lengthy process because the site needs to load the page and do some "work" here or there. Therefore, we can model it by a method, which just waits and does nothing, e.g.:

        public static async Task Manipulate(ChromeDriver driver)
        {
            // Do some useful stuff here instead of just waiting.
            await Task.Delay(60_000);
        }

However, if I start spawning such drivers then very quickly (with under 10 drivers created) some of the created drivers start producing weird errors like: OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:52382/session timed out after 60 seconds.

The test server machine that I am getting these errors has 16 cores and enough RAM to open hundreds of Chrome tabs without any problems, yet a small number of chrome drivers (less than 10) seems not working in parallel.

Does anyone has any ideas how to make many chrome drivers work in parallel? Ideally I'd want to open (3-4X the number of cores) drivers because they will mostly wait and do nothing.

Thanks.

Konstantin Konstantinov
  • 1,363
  • 11
  • 21
  • It is unclear exactly how you are launching the tests. But, to give you some sort of direction, you need to use some sort of pooling of your threads to ensure that you don't stave the machine of resources. [Here is an example in Powershell](https://github.com/apache/lucenenet/blob/f249ca95b47bc7f217479f0f1a779e404ae2e734/build/azure-templates/run-tests-on-os.yml#L111-L197) that runs tests in parallel maximum X processes at a time. The key things to look at are `Get-Job` and `Start-Job`. – NightOwl888 Jan 31 '21 at 06:33
  • C#/.NET just isn't the right tool to run browser-based automated tests. The entire web-dev ecosystem has shifted to using tools like Cypress and Mocha which command headless instances of Chrome running either locally or in a farm. – Dai Feb 03 '21 at 05:50
  • Likely a duplicate of this much more concisely worded question: https://stackoverflow.com/questions/52220159/how-many-maximum-number-of-simultaneous-chrome-connections-threads-i-can-start-t – Ian Mercer Feb 03 '21 at 05:52
  • @IanMercer I agree this mostly qualifies as a duplicate question. Except that the answer was not accepted and the comments hint that you can run multiple browsers, which the code above does. Please, do not that these `http://localhost:52382/session` are different for each driver and it does look that the [not accepted yet] answer in the reference above is not applicable. – Konstantin Konstantinov Feb 04 '21 at 12:46
  • C# is very great for running any number of tests in parallel with no issues. We actually run test against Selenoid deployment. – AlfeG Feb 24 '21 at 13:34

1 Answers1

0

We achieve this using NUnit parallel run, parakkekuzabke by fixture.

Allocate driver during OneTimeSetup. Do whatever test need in single fixture. On OneTimeTearDown, dispose driver. We do this in base class that all web acceptance test fixtures are inherit

    [Parallelizable(ParallelScope.Fixtures)]
    public abstract class WebDriverTest
    {
        protected IDriver driver;

        [OneTimeSetup]
        public void PrepareDriver()
        {
            // ...
            this.driver = new ChromeDriver(chromeDriverService, options);
            // ...
        }

        [OneTimeTearDown]
        public void CleanupDriver()
        {
            this.driver.Dispose();
        }

        [TearDown]
        public void ScreenshotForFailedTest()
        {
            var testStatus = GetTestStatus();

            if (!string.IsNullOrEmpty(testStatus) && testStatus.Equals("Failed"))
            {
                this.driver.TakeScreenshot(); // extension method with a take screenshot functionality
                // log more details if needed
            }
        }
    }

[OneTimeTearDown] is executed even if there is failed tests

As bonus we take screen

Using this snippet we run around 500 smoke tests against chrome in 5-6 minutes on each commit.

AlfeG
  • 1,475
  • 4
  • 18
  • 33