Quick & fast template matching on screen. Coordinates needed too. Java

Question

I need a way to find an image on the screen. I've searched for ways to do this on SO but some take extremely long. I need it to be fast and efficient, does not need to be accurate. Basically i'm planning to compare or search for a small pixelated image, say 11x10 pixels for example, on the screen.

I also need a way to know the x and y coordinates of the small image on the screen.

Although I've looked through many tools out there like JavaCV and OpenCV, I just wanted to see if there are any other ways to do this.

TL;DR

I need a fast way to search for a small (11x10 example.) image on the screen and know its x,y coordinates.

score 1 · Answer 1 · edited May 23 '17 at 11:53

1

I think you many find this answer relevant! But it is for Windows & in c++. But i'm sure that you can convert it very easily to any language.

edited May 23 '17 at 11:53

Community

1
1

answered Jun 02 '15 at 05:39

Balaji R

1,805
22
41

score 0 · Answer 2 · edited May 23 '17 at 12:08

This question is very old, But im trying to acheive the exact same thing here. Ive found that combining these answers would do the trick:

Convert BufferedImage TYPE_INT_RGB to OpenCV Mat Object

OpenCV Template Matching example in Android

The reason you need to do a conversion is because when u grab a screenshot with awt.Robot class its in the INT_RGB format. The matching template example expects bytes and you cannot grab byte data from this type of image directly.

Heres my implementation of these two answers, but it is incomplete. The output is all screwed up and i think it may have something to do with the IntBuffer/ByteBuffers.

-Edit-

I've added a new helper method that converts a INT_RGB to a BYTE_BGR. I can now grab the coordinates of template on the image using matchLoc.This seems to work pretty well, I was able to use this with a robot that clicks the start menu for me based on the template.

private BufferedImage FindTemplate() {                     


    System.out.println("\nRunning Template Matching");
    int match_method = Imgproc.TM_SQDIFF;

    BufferedImage screenShot = null;

    try {
        Robot rob = new Robot();
        screenShot = rob.createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));


    } catch (AWTException ex) {
        Logger.getLogger(MainGUI.class.getName()).log(Level.SEVERE, null, ex);
    }

   if(screenShot == null) return;

    Mat img = BufferedImageToMat(convertIntRGBTo3ByteBGR(screenShot));
    String templateFile = "C:\\Temp\\template1.JPG"; 
    Mat templ = Highgui.imread(templateFile);

    // / Create the result matrix
    int result_cols = img.cols() - templ.cols() + 1;
    int result_rows = img.rows() - templ.rows() + 1;
    Mat result = new Mat(result_rows, result_cols, CvType.CV_32FC1);

    // / Do the Matching and Normalize
    Imgproc.matchTemplate(img, templ, result, match_method);
    Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());
    Highgui.imwrite("out2.png", result);

    // / Localizing the best match with minMaxLoc
    MinMaxLocResult mmr = Core.minMaxLoc(result);

    Point matchLoc;
    if (match_method == Imgproc.TM_SQDIFF
            || match_method == Imgproc.TM_SQDIFF_NORMED) {
        matchLoc = mmr.minLoc;
    } else {
        matchLoc = mmr.maxLoc;
    }

    Graphics2D graphics = screenShot.createGraphics();
    graphics.setColor(Color.red);
    graphics.setStroke(new BasicStroke(3));
    graphics.drawRect(matchLoc.x, matchLoc.y, templ.width(), templ.height());
    graphics.dispose();

 return screenShot;


}                             


private Mat BufferedImageToMat(BufferedImage img){

    int[] data = ((DataBufferInt) img.getRaster().getDataBuffer()).getData();
    ByteBuffer byteBuffer = ByteBuffer.allocate(data.length * 4);
    IntBuffer intBuffer = byteBuffer.asIntBuffer();
    intBuffer.put(data);

    Mat mat = new Mat(img.getHeight(), img.getWidth(), CvType.CV_8UC3);
    mat.put(0, 0, byteBuffer.array());
    return mat;


}`

private BufferedImage convertIntRGBTo3ByteBGR(BufferedImage img){
    BufferedImage convertedImage = new BufferedImage(img.getWidth(), img.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
    Graphics2D graphics = convertedImage.createGraphics();
    graphics.drawImage(img, 0, 0, null);
    graphics.dispose();

    return convertedImage;
}

Results:

Template:

Quick & fast template matching on screen. Coordinates needed too. Java

TL;DR

2 Answers2