How to create an AXUIElementRef from an NSView or NSWindow?

Question

Concerning the macOS Accessibility API, is there anyway to create an AXUIElementRef that corresponds to either an NSView or an NSWindow?

There appears to have been a way of doing this back in the days of Carbon using AXUIElementCreateWithHIObjectAndIdentifier but that function isn't available anymore.

The only method I'm aware of is to use the Accessibility API to recursively search the entire hierarchy of UI elements of your application looking for one that matches the NSView or NSWindow. But in addition to being an onerous solution, it's not even guaranteed to succeed as there might not be a way to positively correspond an AXUIElementRef and a Cocoa object by just using the available attributes of the AXUIElementRef.

I am willing to consider undocumented APIs that help accomplish this.

Answer 1

I found the way to do the same thing in iOS.

I know this isn't a direct answer to your question, but I'll try to explain what I have done to find it in iOS, and hopefully you'll be able to do the same in macOS. Also, this might be useful for other readers...

I started by guessing that the process itself is creating the AXUIElementRef, so it has to create them when I request accessibility attributes that have AXUIElementRef values, such as kAXUIElementAttributeChildren.

I then created an app, and dlsym'ed _AXUIElementCreateAppElementWithPid(int pid), calling it with [[NSProcessInfo processInfo] processIdentifier]. I received the root AXUIElementRef, which I then passed into AXError AXUIElementCopyMultipleAttributeValues(AXUIElementRef element, CFArrayRef attributes, AXCopyMultipleAttributeOptions options, CFArrayRef _Nullable *values), requesting kAXUIElementAttributeChildren, and it worked (should be run on main thread)!

I started debugging the AXUIElementCopyMultipleAttributeValues call carefully into the assembly code, which went pretty much like that (this is very pseudo-code, off course...):

// serialize the arguments for MIG call
if (axUIElementRef->pid == self pid) {
    // that's good that we are calling our own process, we can easily keep debugging!
    _MIGXAAXUIElementCopyMultipleAttributeValues(serialized arguments) {
         // Get the original element from the AXUIElementRef:
         UIView* view = _AXElementForAXUIElementUniqueId(id);
         [view accessibilityAttributeValue:kAXUIElementAttributeChildren] {
              [view _accessibilityUserTestingChildren] {
                   // since this is the UIApplication element, it just gets the windows:
                   NSArray*<UIWindow*> winArr = [(UIApplication*)view _accessibilityWindows];
                   // for each value in result, convert to AX value:
                   ...
                   AXConvertOutgoingValue(winArr) {
                       // For each UIView, convert to AXUIElementRef:
                       AXUIElementRef e = _AXCreateAXUIElementWithElement(winArr[i]);
                   }
              }
         }
    }
} else {
    // Do the same only if we are entitled, and outside our process
}

So, in iOS, you simply call AXUIElementRef _AXCreateAXUIElementWithElement(UIView*); to convert from UI to accessibility element, and UIView* _AXElementForAXUIElementUniqueId(AXUIElementRefGetUniqueID(AXUIElementRef)); in the opposite direction.

All symbols are from AXRuntime.framework.

In mac, you'll need to link against ApplicationServices.framework and try something similar.

Hope this helps...