Microsoft UI Automation/Get The Item That The User Clicks

Question

I am trying to figure out what item (for example document, web page tab, window, picture, folder) the user has clicked on. I started by using the following code when I detect a global left mouse click:

System.Drawing.Point MousePoint = System.Windows.Forms.Cursor.Position;
AutomationElement AutomationElement = AutomationElement.FromPoint(new System.Windows.Point(MousePoint.X, MousePoint.Y));
Console.WriteLine(AutomationElement.Current.Name);

This seems to work well in most conditions. However, I need to (if possible) get names of documents/images/folders inside Windows Explorer for example. The value returned when I click a document in the right hand pane of Windows Explorer (not the tree view) is "Name". Is there anyway to get the actual document name? For some reason, clicking sub-folders in the tree view returns the name of the folder, which is what I want.

I also notice that the code seems to display the document/image/folder name when clicked if the Windows Explorer view is set to icons (medium, large or extra large). Is there any reason why other views return "Name" or empty string while medium, large and extra large icons return the actual document/image/folder name? Is it to do with the size of the object clicked? I could really do with a way round this if possible?

I apologise, I am new to UI Automation and just really want a way to find the name of the object (file, folder, document, picture, web page tab etc.) that the user has clicked on. Any help anyone could give would be great.

Value may be more useful than Name. UIAutomation is not intrinsically aware that the "thing" the user is interacting with represents a file and you are relying on Explorer to behave predictably with its naming which it appears not to do - does it show extensions for know file types? For Explorer there are better ways to query the current selection (ShellWindows) — Alex K., May 07 '15 at 14:38
@AlexK. I cannot find a `AutomationElement.Current.Value` field? — Jonathan, May 08 '15 at 10:38
I edited the tags to this question to remove the [tag:microsoft] tag, which should not be used, as stated in the tag wiki (though its continued existence is problematic in itself); this is further to the Meta question asked here: http://meta.stackoverflow.com/q/293754/82548. I'd also ask you to consider - I'm unfamiliar with c# - whether the [tag:hook] tag adds anything relevant to your question. I suspect it doesn't, but I'm unsure. — David Thomas, May 09 '15 at 13:18

Reza Aghaei · Answer 1 · 2022-04-25T21:08:35.317

You need to listen to InvokePattern.InvokedEvent UI automation event.

Example

The following example assumes you have an open instance of windows "Calculator" app. Then if you run the application, when you click on any button in Calculator, we handle the click event and show what button has clicked.

[DllImport("user32.dll")]
private extern static IntPtr FindWindow(string lpClassName, string lpWindowName);
protected override void OnLoad(EventArgs e)
{
    base.OnLoad(e);
    var proc = System.Diagnostics.Process.GetProcessesByName("Calculator")
                        .FirstOrDefault();
    if (proc != null)
    {
        var mainHwnd = FindWindow(null, "Calculator");
        var mainWndElement = AutomationElement.FromHandle(mainHwnd);
        Automation.AddAutomationEventHandler(
            InvokePattern.InvokedEvent, mainWndElement,
            TreeScope.Subtree, (s1, e1) =>
            {
                var element = s1 as AutomationElement;
                MessageBox.Show($"'{element.Current.Name}' clicked by user!");
            });
    }
}
protected override void OnFormClosing(FormClosingEventArgs e)
{
    Automation.RemoveAllEventHandlers();
    base.OnFormClosing(e);
}

You need to add reference to UIAutomationClient and UIAutomationTypes assemblies.

Note

You can easily extend the example and listen to Win32_ProcessStartTrace to detect when Calculator opens. You can see an example to detect when an external process window closes which is using Win32_ProcessStopTrace.
Another option is using SetWinEventHook you can listen to some events from other processes and register a WinEventProc callback method to receive the event when the event raised. Here, you are interested in EVENT_OBJECT_INVOKED event.

Microsoft UI Automation/Get The Item That The User Clicks

1 Answers1

Linked