1

Given a rectangle that represents an area on a Windows screen that contains text, what is the best way to extract the text?

I know that it is possible using OCR, but even after significant pre processing, the quality is really poor. Getting the Window Text using Win32 API does not always work as well.

Assuming that the text was rendered using a font, is it possible to get it from there? Any directions would be extremely helpful. Thanks!

imriqwe
  • 1,455
  • 11
  • 15
  • That's what OCR is for. What do you mean the quality is poor? If the font is not at all handwriting, OCR gives good results. Getting the Window text using Win32 API will also *never* work if the text is a drawing. – Michael Chourdakis Apr 10 '19 at 13:27
  • OCR accuracy is not perfect. In my case, since the text is rendered using a font by Windows, I wish to extract the text from the underlying font layer, rather then use OCR. – imriqwe Apr 10 '19 at 20:54
  • There is no *underlying font layer*. Once the text has been rendered, it's rendered. OCR is the only way you're going to read it, unless you want to grab a bitmap image of the window and try to do it yourself in your code (AKA reinventing the wheel, as you'd be trying to produce your own OCR system). – Ken White Apr 11 '19 at 00:22
  • I understand that given the current frame, the text is already rendered. But assuming that I want to get the text from a rectangle as rendered in the next rendering - is that something possible? – imriqwe Apr 11 '19 at 07:34
  • Does this answer your question? [Reading from a text field in another application's window](https://stackoverflow.com/questions/352236/reading-from-a-text-field-in-another-applications-window) – StayOnTarget Jun 28 '22 at 12:10

2 Answers2

3

Given a rectangle that represents an area on window screen, the best way to extract text is indeed OCR. Use a better OCR library like this one from Microsoft.

The reason getting the window text using Win32 API does not work well is because there may be multiple windows in that rectangle. You will have to find out what all windows the rectangle contains and send a message to get the text for each window. It is not impossible but difficult to do and even if you manage to do that, you will run into issues of text alignment, etc. OCR is your best option.

Security Guard
  • 414
  • 2
  • 7
  • Thanks for your reply, but I asked specifically about non-OCR options. OCR is not 100% accurate, and I look to achieve this. – imriqwe Apr 10 '19 at 20:53
  • 2
    I gave you a solution for non OCR as well using underlying coordinates to find windows and querying text. If you do not want to implement it, there is no other solution. – Security Guard Apr 11 '19 at 00:40
  • After thinking about this little more, your only option is to use OCR. If an application uses win32k apis, you will be able to get the text out by sending get window text. If application uses any other rendering on the screen, like opengl or directx, you will get nothing except by OCR. – Security Guard Apr 11 '19 at 20:11
0

It does seem possible without using OCR, as NirSoft SysExporter can do this:

https://www.nirsoft.net/utils/sysexp.html

This may be suitable for programmatic use as it can be run from a command line:

Starting from version 1.70, you can export the content of Windows control from command-line, without displaying any user interface.

You may not be able to target it at a specific rectangle on the screen, but maybe the same result could be achieved by first scraping everything followed by some post-processing.

Further basic info:

SysExporter utility allows you to grab the data stored in standard list-views, tree-views, list boxes, combo boxes, text-boxes, and WebBrowser/HTML controls from almost any application running on your system, and export it to text, HTML or XML file.

...

Known Limitations

SysExporter can export data from most combo boxes, list boxes, tree-view, and list-view controls, but not from all of them. There are some applications that use these controls to display data, but the data itself is not actually stored in the control, but in another location in the computer's memory. In such cases, SysExporter won't be able to export the data.

Personally I've used it to grab text from what look like label controls.

StayOnTarget
  • 11,743
  • 10
  • 52
  • 81