0

I want to get the word under mouse cursor using OCR, mainly form web pages and ebooks, my procedures are:

  1. Get the location of the rectangle containing the word under mouse cursor. For example , if my mouse cursor is on century , then I need to get the rectangle containing century enter image description here

  2. Take screen shot of the rectangle

  3. Using OCR engines like Tesseract to convert the rectangular image to text, then I get the word under mouse cursor this way:)

The difficulty resides in the first step , I know nearly nothing about image processing , so cannot figure out a way to locate the rectangle . Speed matters in my scenario , so please guide me a fast and efficient solution on this ? Thanks so much!

iMath
  • 2,326
  • 2
  • 43
  • 75
  • Could you elaborate on why the first step involves image processing? My first choice would probably be to get the cursor position (e.g. with this library: https://github.com/PyUserInput/PyUserInput) and then perform OCR on the area surrounding it, choosing the rectangle that contains the cursor (or closest to it if no such rectangle exists). – andersource Dec 22 '18 at 21:03
  • @andersource My concern is mainly on image processing, so I skipped the cursor position step; Since speed matters in my scenario , so I want to locate the rectangle under mouse cursor directly instead of "perform OCR on the area surrounding it, choosing the rectangle that contains the cursor". BTW, how do you determine the size of the area surrounding the cursor, since text size varies , so I think a fixed size might not be a good solution . – iMath Dec 23 '18 at 04:32
  • There are several ways I can think of doing this, but it depends on how much variability you really expect to see in text size and location. I still don't quite understand how you would like to use image processing to find the rectangle? Do you mean find the text rectangle based on the cursor position? Or find a rectangle of mouse-selected text? – andersource Dec 23 '18 at 07:53
  • @andersource Thanks ! 1) I mean find the text rectangle based on the cursor position, see the updated image above ; 2)I want to get word under mouse cursor mainly form web pages and ebooks, the English text is horizontal and I think the maximal size of a target rectangle might be of 350px width and 50px height; – iMath Dec 23 '18 at 09:19
  • Why don't you just cut a reasonably big region around the clicked position, run through an OCR (e.g., Tesseract) and get the closest word, given that the OCRs usually give the position of the words as well? – Berriel Dec 23 '18 at 10:19
  • @Berriel This is what I did with a rectangle of 210px width and 50px height , the problemwith this wayis it isn't fast, so I am trying to get the location of the rectangle containing the word under mouse cursor, assuming that a smaller rectangle would speed up the text-getting process. – iMath Dec 23 '18 at 12:41
  • So if I understand correctly you want to classify where there's text closest to the cursor without performing OCR? [It's possible](https://stackoverflow.com/questions/37771263/detect-text-area-in-an-image-using-python-and-opencv) but I'm not sure if the performance will be significantly different – andersource Dec 23 '18 at 13:26

0 Answers0