2

I am struggling to find a reliable way to get the content/text of the window that is currently in the foreground. It should be able to determine the text from every possible program that a user is currently using, if possible

What I tried:

  • Take a screenshot of the currently active window, apply some filters and run an OCR algorithm (tesseract .Net wrapper). This works, but takes a long time and is not very accurate.
  • Then I tried some Windows API functions (FindWindow and SendMessage), as described here. I could make it run for the standard Editor (notepad) for example, but not for most other programs
  • I also tried to make it work with AutoHotKey and the WinGetText function and again a .Net Wrapper. Here, I just get some info about the window, but in no way the text of it...

Unfortunately, now, I don't have any other idea what to do as I am stuck in every way... Does someone have experience with this or knows a way that works? Any suggestion is really much appreciated

Community
  • 1
  • 1
casaout
  • 1,819
  • 3
  • 24
  • 54
  • 2
    There's no generic way to do this. If a program supports accessibility then you can probably use the accessibility API to extract the text. If the text is the caption of a control (e.g. a STATIC) then you can use `GetWindowText`. If it's simply plain text that's rendered via `TextOut` or similar then OCR is your only option. – Jonathan Potter Apr 22 '15 at 12:10
  • UIAutomation is the standard way. Not all apps expose themselves that way though. – David Heffernan Apr 22 '15 at 12:23
  • @JonathanPotter: Thanks for your comment! Yeah the text usually might not be the content of a control, just from the IDE, the browser-window, Word, Outlook, etc. Do you maybe know if it's possible to hook somehow to the text rendering (i.e. where the actual window is created)? – casaout Apr 22 '15 at 12:40
  • @DavidHeffernan: Thank you! Do you maybe have a link to get started? (is it something like this https://msdn.microsoft.com/en-us/library/ms788751(v=vs.110).aspx) Thanks! – casaout Apr 22 '15 at 12:40
  • 1
    @casaout yes, that should work; [here are](http://blogs.msdn.com/b/oldnewthing/archive/2013/04/08/10409196.aspx) [more resources](http://blogs.msdn.com/b/oldnewthing/archive/2015/02/16/10593625.aspx) – andlabs Apr 27 '15 at 00:50

1 Answers1

0

It will be difficult to find a single solution to retrieve text from applications. Different methods for different programs will be required.

For AutoHotkey, AccViewer, which makes use of Acc.ahk is the best method of first resort. Acc works on a large variety of controls and also elements within controls, it can cover far more control types than AutoHotkey's ControlGet command.

Acc Library [AHK_L] (updated 09/27/2012) - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/77303-acc-library-ahk-l-updated-09272012/

Accessible Info Viewer - Alpha Release (2012-09-20) - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/77888-accessible-info-viewer-alpha-release-2012-09-20/

A link describing some further text retrieval methods:

AutoHotKey ControlGet

Note also:

COM (Component Object Model), is handled natively by AutoHotkey. It can be used to retrieve the text from web elements in Internet Explorer, and via VBA code, text can be retrieved from MS Office programs such as MS Excel and MS Word.

Community
  • 1
  • 1
vafylec
  • 965
  • 1
  • 6
  • 23