4

I want to be able to grab all the text from the frontmost window on my screen, pipe it to a url parser and choose which url to open from the parsed text.

To do this I would need a way to access all textual contents of the frontmost window. I've come close by using applescript but I can't find a way to get the actual text. I'm suspecting that swift might be able to help, but I'm lost as to how to even google this, as most of my searches end up finding only trivial approaches like getting the url from a tab in chrome etc..

This is the gist of what I want:

tell the frontmost application
    get contents of frontmost window as text
end tell
AstroCB
  • 12,337
  • 20
  • 57
  • 73
brujoand
  • 815
  • 8
  • 15
  • 1
    `I can't find a way to get the actual text` I wonder what text you're talking about. Apps can have very different ways of displaying text. And if they don't expose it on purpose, there's no way to get it from your own app. – Eric Aya Jul 30 '16 at 10:08
  • I guess this is what I feared. My hope was that since voice over seems to work on most apps there would also be a way for me to get the text that voice over uses. – brujoand Jul 30 '16 at 10:11
  • Well I should have said "there's no way *that I can think of right now*"... maybe someone will surprise us. Also, your comment is interesting: maybe you can use Accessibility to grab some exposed text, as Voice Over does? I don't know, actually. – Eric Aya Jul 30 '16 at 10:14
  • Interesting: http://stackoverflow.com/q/6544311/2227743 – Eric Aya Jul 30 '16 at 10:34

2 Answers2

1

Have you tried to get all UI elements of the specific application?

set appname to "System Preferences" -------------------------- Set this to the App you want to look at

display dialog "Set the app you want to look at" default answer "System Preferences" buttons {"OK"} default button 1
set appname to text returned of result

set winstuff to "defaultval"
set menustuff to "defaultval"

tell application appname
    activate
end tell

delay 0.5

tell application "System Events"
    tell process appname
        set winstuff to entire contents of front window
        set menustuff to entire contents of menu bar 1
    end tell
end tell
--return winstuff & "\r\r\r\r" & menustuff -- comment this out to get just winstuff
return winstuff -- comment this out too to get just menustuff
--return menustuff
oa-
  • 361
  • 1
  • 3
  • 14
  • Thanks, I did try this. But it is unpredictable which applications this will work for and not unfortunately. – brujoand Jan 24 '17 at 07:36
0

A solution for JXA:

//Example get all text from "ScreenFlow"
var sProc  = "ScreenFlow"
var system = Application('System Events')
var elements = system.processes[sProc].windows[0].entireContents()
var s = ""
for (var i=0;i<elements.length;i++){
    s = s + elements[i].name() + ","
}
s = s.replace(/null,/g,"")

//Return s
console.log(s)

There are probably other element children you can add to make this list more information dense though. An example output from my video editor:

"Video,+ Action,Color:,Offset:,Blur Size,Shadow,Opacity,Scale:,Z Rotation:,Reflection,Y Rotation:,Opacity:,Cropping,X Rotation:,Position:,Color Controls,Corner Round:,Video Filters,Duration: 1 min 0 secs,UI Mass Email App,UI Mass Email App,"

Note this doesn't grab menu text. For that you will have to loop through system.processes[sProc].menuBars

Sancarn
  • 2,575
  • 20
  • 45