1

Scenario: I have one more more WPF applications opened. I cannot change the source code of these apps. My purpose is Whenever user clicks any control (button, combobox, textbox etc.) on these apps, I want to know which control / element is clicked, and log it. Simply obtaining the name of the element would be enough like so: "App1 - button3 is clicked." or "App2 - button1 is clicked". If possible, I want to achieve this for both Winforms and WPF apps, but WPF is more important.

  • Any way to do accomplish this in the background is OK.
  • I tried examining and using source codes of snoopwpf (Since it is able to detect the element under the mouse cursor by pressing CTRL+SHIFT), but I wasn't able to achieve my purpose. I could not get the elements in different AppDomains. (AppDomainHelper.GetAppDomains() also returns null)
  • I looked a little into pywinauto module, however couldn't find such a functionality.
klavre moboard
  • 27
  • 1
  • What you want is probably [UI Automation](https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview), but you might have to jump through extra hoops if the WPF app didn't name their buttons or controls. Meaning, you might have to figure out the buttons from screen coordinates or unique ids or other means. – Tam Bui Apr 25 '22 at 15:14
  • @TamBui Thanks, but I am not sure if System.Windows.Automation is the solution for my problem. With it, I can get the AutomationElement object by providing a name, but I want the other way around: I want to get the name of the clicked object. Does UI Automation provide this? If so, can you please provide an example code or links? Thanks. – klavre moboard Apr 25 '22 at 15:30
  • You can use [SetWinEventHook](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwineventhook), to receive notifications for the [EVENT_OBJECT_INVOKED](https://learn.microsoft.com/en-us/windows/win32/winauto/event-constants) + `EVENT_OBJECT_FOCUS` and `EVENT_OBJECT_STATECHANGE` -- Can be used in combination with UIAutomation methods. – Jimi Apr 25 '22 at 16:15
  • 2
    @klavremoboard I don't have any examples, but here are some useful links. Essentially you would have to inject an event listener to the UIAutomation button that you want to wait to be clicked. [Link 1](https://stackoverflow.com/questions/33164659/how-to-capture-send-button-event-for-outlook-using-ui-automation-in-c) [Link 2](https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-events-overview) – Tam Bui Apr 25 '22 at 17:02
  • pywinauto's implementation of event handlers is very draft, but you can take a look: https://github.com/pywinauto/pywinauto/pull/701 – Vasily Ryabov Apr 25 '22 at 18:32
  • 1
    @Jimi I tried WinEventHook too, but couldn't make it work. I tried adding all 3 of the events you mentioned but there doesn't seem to be a difference. The event function is not being called after being hooked. I also get System.ExecutionEngineException after some time running the app and program crashes. Here is how I used SetWinEventHook (simplified code): https://pastebin.com/3CXUr5a6 . is there something I did wrong, How should I use it for my purpose? If you could elaborate in the context of my question, it would be great. Thanks. – klavre moboard Apr 25 '22 at 19:35
  • You're overwriting the pointer to the Hook Proc and your delegate must be protected with `GCHandle.Alloc()` or a SafeHandle, otherwise will be garbage collected (i.e., don't set `new WinEventDelegate(...)` in the call to `SetWinEventHook()`). - The flags must be set to `WINEVENT_OUTOFCONTEXT | WINEVENT_SKIPOWNPROCESS` - A couple of `int` must be `long` instead -- You may refer to [this answer](https://stackoverflow.com/a/48812831/7444103) for the declarations and a sample setup -- I'll give it a look when I have some free time. – Jimi Apr 25 '22 at 20:50
  • You can take a look at this working example which uses UI automation: [Microsoft UI Automation/Get The Item That The User Clicks](https://stackoverflow.com/a/72005695/3110834) or this one which contains two example using UI Automation and using Windows event hook: [Detect when a specific window in another process opens or closes](https://stackoverflow.com/a/40285043/3110834) – Reza Aghaei Apr 25 '22 at 21:15
  • @TamBui Thank you, this actually helped. However, when I use `InvokePattern.InvokedEvent` for the `AddAutomationEventHandler` method, only button clicks are listened. I want to listen all clicks the user makes such as opening and selecting an element from a combobox, checking a checkbox, clicking on a radio button, etc. `SelectionItemPattern.ElementSelectedEvent` seems like it solves the combobox problem but the others remain. Also I want to know if the user clicks on other "non invokable" controls too If possible (such as label, textbox etc). Is there a way to do this? Thanks. – klavre moboard Apr 26 '22 at 14:47
  • 1
    I'm not an expert on UI Automation and haven't used it in ten years, but I believe you can find further information from [a link like this](https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-support-for-the-checkbox-control-type) that describes the control type capabilities of just this one control, but if you look at the navigation menu on the left, it shows pages for various control types like combobox, checkbox, calendar, etc. Good luck! – Tam Bui Apr 27 '22 at 05:43
  • @Jimi Thanks for the example. I used the `Hook` and `NativeMethods` classes directly, as you provided in the example. And [here](https://pastebin.com/8GwDyZnu) is my driver code (simplified). (I use a list of events too see all at the same time but I tried them seperately too.). I have a few questions: Most importantly, how can I obtain the control object or at least its name & ID with this event callback? Secondly, `EVENT_OBJECT_INVOKED` is never called, shouldn't it be called when I click the buttons? (STATECHANGE is fine). Is there any other event type I should try for my purpose? Thanks. – klavre moboard Apr 28 '22 at 14:44
  • You need to post your code here, don't post a link to different site. -- Note that you can hook a range of events using the same delegate; don't try to create more than one delegate, unless you're prepared to handle them all separately. – Jimi Apr 28 '22 at 17:14

0 Answers0