How to manipulate Chrome extension's widget with Selenium or another tool?

Question

I've been looking for a way to click on an extension's widget outside the HTML DOM with Selenium in C# (like to turn it on or so...), but with no success. I've tried the addExtension method, but it crashes Chrome and tried using the default profile so the active extensions will load with the ChromeDriver instance, but that didn't work for me either.

These elements don't have a page I can navigate to in order to change their settings (They have a pop-up when clicking the widget), so I need another way to do this.

Do any of you Selenium experts know if I can achieve it with Selenium or maybe it's a feature to be implemented in the future?

Or maybe there's another good free tool to do this?

I believe that such a thing is not possible. I'm happy to be proven wrong though. — Rob W, Aug 14 '13 at 16:39
I know it's probably not the solution you intended as this doesn't return the handle to the button, but it's a good work around for now... — Moshisho, Jan 29 '14 at 15:12

Moshisho · Accepted Answer · 2014-02-13T16:40:43.347

I found this image search dll library with a wrapper in AutoIt that solved it for me. It's not the most robust solution you'll find, but for most people it's working great. It worked for me with Chrome's non identifiable controls.

Very simple use:

$result = _ImageSearchArea("ButtonImagePath.png", $tolerance, $Left, $Top, $Right, $Bottom, $x, $y, 100)

Where $x and $y are the result location so you can just send a click afterwards. Getting the image was very easy too with snipping tool. More info here and here.

Edit: If you want to use it in C# it's possible too with interop with it's dll: (It's really useful for automating mouse clicks or so on unidentified buttons)

class ImageHelper
{
    [DllImport("ImageSearchDLL.dll")]
    private static extern IntPtr ImageSearch(int x, int y, int right, int bottom, [MarshalAs(UnmanagedType.LPStr)]string imagePath);

    private static String[] UseImageSearch()
    {
        int right = Screen.PrimaryScreen.WorkingArea.Right;
        int bottom = Screen.PrimaryScreen.WorkingArea.Bottom;

        IntPtr result = ImageSearch(0, 0, right, bottom, "imageFileName.png");
        String res = Marshal.PtrToStringAnsi(result);


        if (res[0] == '0') return null;//not found

        String[] data = res.Split('|');
        int x; 
        int y; 
        int.TryParse(data[1], out x);
        int.TryParse(data[2], out y);

        //0->found, 1->x, 2->y, 3->image width, 4->image height
        Cursor.Position = new Point(x, y);

        return data;
    }
}

score 0 · Answer 2 · answered Jun 24 '15 at 10:31

UIAutomation is probably the best solution to this case. Chrome has an extensions UI element 'Container' which inside it you can find the extensions's widgets as buttons. This could be verified with Visual UI Automation Verify:

enter image description here

Related code will be:

AutomationElement chrome = AutomationElement.RootElement.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.ClassNameProperty, "Chrome_WidgetWin_1"));
AutomationElement extensionsContainer = chrome.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.NameProperty, "Extensions"));

After clicking the widget, it'll be hard to find elements related to the extension if they are still outside the DOM, so in that case I think the image search (with Sikuli or like in the other answer) is a good try.

How to manipulate Chrome extension's widget with Selenium or another tool?

2 Answers2

Linked