Executing Javascript elementFromPoint through Selenium driver

Question

I am trying to implement an "object picker" to my Selenium based framework as is common on most commercial automation tools. To do this I am using a Javascript command to find the element at the mouse location, but I am not getting the element I expect.

If I am using ChromeDriver or InternetExplorerDriver the script always returns the header object. No matter what webpage I look at or the position of the mouse. Although it sounds like the script is taking the coordinates 0, 0 instead of the mouse position I have confirmed that Cursor.Position is sending the correct values.

If I am using FirefoxDriver I get an exception:

"Argument 1 of Document.elementFromPoint is not a finite floating-point value. (UnexpectedJavaScriptError)"

Can anyone see what I am doing wrong?

    private void OnHovering()
    {
        if (Control.ModifierKeys == System.Windows.Forms.Keys.Control)
        {
            IWebElement ele = null;
            try
            {
                // Find the element at the mouse position
                if (driver is IJavaScriptExecutor)
                    ele = (IWebElement)((IJavaScriptExecutor)driver).ExecuteScript(
                        "return document.elementFromPoint(arguments[0], arguments[1])", 
                        new int[] { Cursor.Position.X, Cursor.Position.Y });

                // Select the element found
                if (ele != null)
                    SelectElement(ele);
            }
            catch (Exception) { }
        }
    }

Thanks!

With the Mouse ;-) the function occurs within a Timer tick. If the user is holding ctrl, the element under the mouse gets selected — Scotty, Aug 10 '15 at 02:36
I am not the developer of the webpage, but I tried many different webpages... www.nab.com.au, www.microsoft.com, www.google.com. No matter what the site is I get the header returned. Cursor.Position.X and Cursor.Position.Y are the correct values, it just seems that I am not passing them correctly to the "ExecuteScript" method. — Scotty, Aug 10 '15 at 02:40
Yeah, thanks I've just reproduced the problem using Python bindings and SO site as a target - no matter what the coordinates are, the header element is returned. — alecxe, Aug 10 '15 at 02:42

alecxe · Accepted Answer · 2015-08-10T02:53:11.680

4

It's actually about how you are passing the coordinates into the script. Script arguments has to be specified separately as separate ExecuteScript() arguments. What was happening in your case is that you have basically specified one x argument which made it think that y should be considered a default 0 value. And at y=0 there is usually a header.

Instead of:

ele = (IWebElement)((IJavaScriptExecutor)driver).ExecuteScript(
                        "return document.elementFromPoint(arguments[0], arguments[1])", 
                        new int[] { Cursor.Position.X, Cursor.Position.Y });

You should do:

ele = (IWebElement)((IJavaScriptExecutor)driver).ExecuteScript(
                        "return document.elementFromPoint(arguments[0], arguments[1])", 
                        Cursor.Position.X, Cursor.Position.Y);

edited Aug 10 '15 at 02:53

answered Aug 10 '15 at 02:48

alecxe

462,703
120
1,088
1,195

Thanks alecxe, you're right that this was the problem. But the documentation for C# is `object IJavaScriptExecuter.ExecuteScript(string script, params object[] args)`. Is the documentation wrong or am I reading it wrong? To me that means that the second parameter should be an array of arguments. – Scotty Aug 10 '15 at 03:06
@Scotty yeah, this is quite confusing I agree. – alecxe Aug 10 '15 at 04:35

Elmue · Answer 2 · 2015-09-14T21:59:48.293

The code from alecxe works in most cases, but it will fail if the page contains frames or iframes.

Much more code is required to respect also frames / iframes.

/// <summary>
/// Get the element at the viewport coordinates X, Y
/// </summary>
static public RemoteWebElement GetElementFromPoint(RemoteWebDriver i_Driver, int X, int Y)
{
    while (true)
    {
        String s_Script = "return document.elementFromPoint(arguments[0], arguments[1]);";

        RemoteWebElement i_Elem = (RemoteWebElement)i_Driver.ExecuteScript(s_Script, X, Y);
        if (i_Elem == null)
            return null;

        if (i_Elem.TagName != "frame" && i_Elem.TagName != "iframe")
            return i_Elem;

        Point p_Pos = GetElementPosition(i_Elem);
        X -= p_Pos.X;
        Y -= p_Pos.Y;

        i_Driver.SwitchTo().Frame(i_Elem);
    }
}

/// <summary>
/// Get the position of the top/left corner of the Element in the document.
/// NOTE: RemoteWebElement.Location is always measured from the top of the document and ignores the scroll position.
/// </summary>
static public Point GetElementPosition(RemoteWebElement i_Elem)
{
    String s_Script = "var X, Y; "
                    + "if (window.pageYOffset) " // supported by most browsers 
                    + "{ "
                    + "  X = window.pageXOffset; "
                    + "  Y = window.pageYOffset; "
                    + "} "
                    + "else " // Internet Explorer 6, 7, 8
                    + "{ "
                    + "  var  Elem = document.documentElement; "         // <html> node (IE with DOCTYPE)
                    + "  if (!Elem.clientHeight) Elem = document.body; " // <body> node (IE in quirks mode)
                    + "  X = Elem.scrollLeft; "
                    + "  Y = Elem.scrollTop; "
                    + "} "
                    + "return new Array(X, Y);";

    RemoteWebDriver i_Driver = (RemoteWebDriver)i_Elem.WrappedDriver;
    IList<Object>   i_Coord  = (IList<Object>)  i_Driver.ExecuteScript(s_Script);

    int s32_ScrollX = Convert.ToInt32(i_Coord[0]);
    int s32_ScrollY = Convert.ToInt32(i_Coord[1]);

    return new Point(i_Elem.Location.X - s32_ScrollX,
                     i_Elem.Location.Y - s32_ScrollY);
}

This should be implemented in the WebDriver.

score 0 · Answer 3 · answered Nov 22 '19 at 17:37

0

I was having this issue as well. The reason it was being thrown I found out was because the element I was trying to get the location of was hidden (had the attribute display:none). And therefore didn't have a location.

answered Nov 22 '19 at 17:37

Creoix

23
3

Executing Javascript elementFromPoint through Selenium driver

3 Answers3

Linked