2

Imagine there is an application that has a window with a fixed size, where all controls remain at exactly known places (yes, there are such applications). Now imagine that said application's window is quite small and almost unusable on a touch-screen device. My idea now is to build a simple WPF UI, that maps certain controls of the external application to a touch-friendly UI.

(And this is actually not about gaming, but rather about making music. I want to use the small external application live on stage...)

I found a whole bunch of ways of controlling external applications, most of them getting an handle to the external application's window, then getting a handle to a button, then performing a mouse click on that, like in answers provided to the following questions:

Other answers propose to move the mouse to the desired location and then perform certain mouse events:

All of these approaches have drawbacks: Sometimes they need to know the target applications's structure. Sometimes they use deprecated API methods. Sometimes the move the cursor of the mouse to the external application.

So here is my question: How can I perform mouse actions on external applications relative to the external application's window without actually moving the mouse pointer?

Ideally there would be no need for external application's window to be on top, meaning that ideally my WPF UI would be a fullscreen one.

NB: Some commenters suggested using UI Automation for that task. I tried to walk the UI tree of the external application using Inspect, but that yielded unsatisfying results:

enter image description here

Community
  • 1
  • 1
Spontifixus
  • 6,570
  • 9
  • 45
  • 63
  • Yes, there is a way http://geekswithblogs.net/danielggarcia/archive/2014/01/13/simulation-of-mouse-and-keyboard.aspx – netaholic Sep 30 '15 at 12:44
  • Althought it brings the application window on top – netaholic Sep 30 '15 at 12:49
  • @netaholic I read that article. Two things: For once it states that we "move the pointer" indicating that we actually move the pointer, instead of just performing a click at a location. The other thing is that the `mouse_event` API method is deprecated... – Spontifixus Sep 30 '15 at 12:52
  • I'm sure you could, but I wouldn't. UI Automation is the most stable and consistent way of controlling an external program, and you only need to look through the entire control tree once to figure out the structure. Anything else probably won't work all that consistently. – theB Sep 30 '15 at 12:54
  • @theB I don't know which technology the external application is build on, so I have no idea on how to actually get to the control tree of that application. Spy++ only displays the main window of the application. – Spontifixus Sep 30 '15 at 13:04
  • 2
    It's amazing how often UI Automation is overlooked – David Heffernan Sep 30 '15 at 13:20
  • 1
    There are [tools](https://www.google.com/search?q=tool+view+ui+automation+tree&ie=utf-8&oe=utf-8) available for UI automation that allow you to browse the automation tree, which is different from the tree that Spy++ will show you. – theB Sep 30 '15 at 13:26
  • @theB Please see my updated question. The UI Automation tree actually is not that different from the Spy++ one in my case actually... – Spontifixus Sep 30 '15 at 14:00
  • 1
    There's only the native window tree, or the UI Automation (or MSAA) tree. If neither one provides the information you need, you're on your own. There is no other facility, that allows you to inspect interaction points in an unknown application. The only other option would be [SendInput](https://msdn.microsoft.com/en-us/library/windows/desktop/ms646310.aspx), but that requires, that the target application has input focus. Everything else is way too unreliable, to use it on a live stage. – IInspectable Sep 30 '15 at 17:57

0 Answers0