2

The page has a div that appears randomly, that must be clicked quickly or it disappears. I have a try catch inside a while loop looking for it in case the element is not found, the catch block prevents the program from stopping and iterates again looking for it.

I've set the implicit wait to 0, so the driver does not wait if the button is not found. The while loop can iterate about every 3ms and I changed the FindElement method from XPath to CssSelector, but its still too slow.

The div has no id, name, or class. there is only the tagname div, attribute ng-show and the div text to go by. the program DOES find the div but it misses it alot before finally clicking it, so im guessing the while loop is just iterating too slow.

What other bottleneck am I missing, how do I make this iterate as fast as possible?

    private static void FindButtonAndClick()
    {
        driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromMilliseconds(0);

        try 
        {
            while (pageStatusOnline == true)
            {
                if (programStopped == true) { return; };
                try
                {
                    driver.FindElement(By.CssSelector("div[ng-show='!buttonClicked']")).Click();
                    pageStatusOnline = false;
                    buttonClickedAt = "Page1";
                    break;
                } catch { };
            };
        }
        catch 
        {
            Console.WriteLine("Error during FindButtonAndClick()");
            programStopped = true;
        };
    }

Edit: I did some execution time testing and the above code executes in between 11ms-36ms, I'm guessing the fluctuation is because of the flood of exceptions @Dai mentioned.

@Dai mentions that Events are better so Here is my attempt at using Selenium C# Explicit wait on a separate thread:

    bool checkingForButton;
    bool programStopped;
    string buttonPath = "By.CssSelector(div[ng-show='!buttonClicked'])";
    private static void FindButtonAndClick()
    {
        try
        {
            driverPage.Manage().Timeouts().ImplicitWait = TimeSpan.FromMilliseconds(0);
            CancellationTokenSource closeThreadSource = new CancellationTokenSource();
            CancellationToken token = closeThreadSource.Token;
            Thread buttonWaitThread = new Thread(() => WaitForButton(token));
            buttonWaitThread.IsBackground = true;
            buttonWaitThread.Start();
            checkingForButton = true;

            while (checkingForButton == true)
            {
                //WaitForButton on the other thread 
                //will flip checkingForButton
                //when the button is clicked
                //and this while loop will end

            };
            closeThreadSource.Cancel();
            buttonWaitThread.Join();
            closeThreadSource.Dispose();
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error during FindButtonAndClick(): {ex}");
        };
    }

    private static void WaitForButton(CancellationToken token)
    {
        while (!token.IsCancellationRequested)
        {
            if (programStopped == true) { return; };
            WebDriverWait pageWait = new WebDriverWait(driverPage, TimeSpan.FromHours(1));
            pageWait.PollingInterval = TimeSpan.FromMilliseconds(0);
            pageWait.IgnoreExceptionTypes(typeof(NoSuchElementException), typeof(StaleElementReferenceException));
            pageWait.Until(ExpectedConditions.ElementToBeClickable(buttonPath))
            driverPage.FindElement(buttonPath).Click();
            checkingForButton = false;
        }
    }

next step is using System.Diagnostics.StopWatch() to see if waiting for the event instead of hammering the page is in fact faster. Also looking into the MutationObserver alternative.

JNDEV
  • 23
  • 5
  • `} catch { };` <-- Don't do that. Change your code to prevent exceptions from being thrown in the first place. Exceptions wreck performance. – Dai Apr 25 '23 at 00:15
  • A better idea would be to add a DOM `MutationObserver` to the document _immediately_ as the `` element is added and then use that to detect the transient element you're after instead of hammering the DOM in a loop. – Dai Apr 25 '23 at 00:17
  • @dai Some sample code as an answer would be great. I'd like to see it as well. – JeffC Apr 25 '23 at 04:25
  • @JeffC Done - now you owe me an effortpost – Dai Apr 25 '23 at 08:32
  • 11-36 milliseconds is not much time. Are you sure this is the performance bottleneck? Are you sure there *is* a performance problem? C# has a [Stopwatch](https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.stopwatch) class so you can measure the execution time. Do that first, then you will understand the bottleneck. You might also have a performance problem on the web page itself. – Greg Burghardt Apr 25 '23 at 18:16
  • @GregBurghardt You're right its not much, but the question is how to optimize it even further if possible, because many times the button appears and dissapears even faster than the attempt to click it iterates. I've tested finding the button with a Chrome extension and it finds the button every time. and yes I used the System.Diagnostics.Stopwatch to measure the performance. – JNDEV Apr 25 '23 at 19:56

1 Answers1

2
  • I'm not overly familiar with Selenium, WebDriver, and CDP - so I don't know if Selenium does already have a built-in way for programs to interact with the DOM directly (like we did with MSHTML.dll's IHTMLDocument7 before Chromium took-over and required us to do everything via giant-literal-strings-of-JavaScript in our programs, le sigh.

ANYWAY

So the outline of my thought-process and idea initially posted as a comment above, is something like this:

  • Polling is bad (and I don't mean politically): if you ever find yourself writing a loop that tries something then waits, then tries again, then you're probably doing something wrong (excepting spinlocks and embedded/vintage/etc stuff, of course).
  • Events are better: so how can we get an event in our C# program that notifies us exactly when an element is added to the DOM? If we get an event then we can instantly engage and interact with it all within a couple of browser render frames (hopefully) of the event. The answer is that we can by using a MutationObserver in an in-page <script> that can notify us, or instantly perform some action on the DOM within the same script, all whenever a particular HTML element is added or removed from the DOM - or if this element you're after isn't actually added or removed from the DOM, but instead always exists and simply has its class="" or style="" attributes changed, then the MutationObserver can handle that too.
  • Actually I'm not sure how a page within Selenium can send a message back to the host program - but I assume you know how to do that.
  • Moving on: it won't be of much use to insert that <script> after the page's HTML has already started loading - we'll want to insert it before the document even starts loading - (don't worry, browsers handle <script> elements located before the opening <html> just fine). We can do this with the CDP command Page.addScriptToEvaluateOnNewDocument (more here)
  • soooo...
    • I'm unable to run or test this myself, so I've no idea if it works or not :)

The injectable <script>

var mobyDick = null;

const capnAhab = new MutationObserver( ( changes, bahAnpac  ) => {
    for( const c of changes ) {
        const w = tharSheBlows( c );
        if( w ) window.mobyDick = w;
    }
});

function harpoon( c ) {
    if( c.type === 'attributes' && c.attributeName === 'ng-show' ) {
        const el = c.target; // For 'attributes', `c.target` will be a HTMLElement object.
        return el.tagName === 'div' && el.getAttribute('ng-show') === '!buttonClicked' ) ;
    }

    return false;
}

capnAhab.observe( /*target:*/ document.documentElement, /*options:*/ { subtree: true, childList: true, attributes: true, attributeFilter: [ 'ng-show' ] } ); // <-- Surprisingly there is no way to filter by element name, only by attribute name.

function tharSheBlows( c ) {
    if( harpoon( c ) ) {
        clickTheThingie( c.target );
        return c.target;
    }
    else if( c.type === 'childList' && c.addedNodes.length > 0 ) {
        for( const el of Array.from( c.addedNodes ).filter( n => n instanceof HTMLDivElement ).filter( div => div.getAttribute('ng-show') === '!buttonClicked' ) ) {
            clickTheThingie( el );
            return el;
        }
    }

    return null;
}

function clickTheThingie( el ) {
    
    // TODO: Send message to Selenium host:
    // On a lark, I asked ChatGPT and it said I could use `window.external.notify("message")` to send a message to the Selenium host - but when I tried to look it up I saw this open feature-request for it in Edge - so I guess not? https://github.com/MicrosoftEdge/WebView2Samples/issues/43
    // You could use a `debugger;` breakpoint statemewnt or `console.log` - I guess?

    if( 'click' in el ) { el.click(); }

    window.alert( "Towards thee I roll" );
}

C# Selenium

using OpenQA.Selenium.etc...;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.DevTools;
using OpenQA.Selenium.DevTools.V96;
using OpenQA.Selenium.DevTools.etc...;

//

const String THE_PEQUOD = @"
<< COPY THE JAVASCRIPT FROM ABOVE INTO HERE >>
";

//

IDevTools       devTools = (IDevTools)driver;
DevToolsSession session  = devTools.GetDevToolsSession();

var domains = session.GetVersionSpecificDomains<DevToolsSessionDomains>();

domains.Page.Enable( new Page.EnableCommandSettings() );
domains.Page.AddScriptToEvaluateOnNewDocument( new AddScriptToEvaluateOnNewDocumentCommandSettings()
{
    Source = THE_PEQUOD
} );

driver.Navigate().GoToUrl( "https://leekspin.com/" );
Dai
  • 141,631
  • 28
  • 261
  • 374
  • This is great. Thanks! I don't know if I'll ever need it but it's nice to know it exists if I ever do. – JeffC Apr 25 '23 at 14:00