1

I'm looking to make a program that once run, will continuously look for an template image (stored in directory of program) to match in realtime with the screen. Once found, it will click on the image (ie the center of the coords of the best match). The images will be exact copies (size/color) so finding the match should not be very hard.

This process then continues with many other images and then resets to start again with the first image again but once I have the first part working I can just copy the code.

I have downloaded the OpenCV library as it has image matching tools but I am lost. Any help with writing some stub code or pointing me to a helpful resource is much appreciated. I have checked a lot of the OpenCV docs with no luck.

Thanks you.

Mo Beigi
  • 1,614
  • 4
  • 29
  • 50

2 Answers2

1

If you think that the template image would not be very different in the current frame then you should use matchTemplate() of openCV. Its very easy to use and will give you good results.

Have a look here for complete explanation http://docs.opencv.org/doc/tutorials/imgproc/histograms/template_matching/template_matching.html

skm
  • 5,015
  • 8
  • 43
  • 104
  • 2
    addition about speed: complexity of template matching should be linear in the number of templates and their sizes, so if he uses many different big templates, it might be too slow. If the templates have some describeable texture he could try feature matching (SIFT/SURF/ORB for example) where the image is only once processed and each template compared to the features. – Micka Feb 14 '14 at 09:22
  • Yes i've had a look at that page. However, I was unable to use the screen as the source (so that template images could be matched with it). Is one solution to take a screenshot of the screen every second and use that or does the library have built in solutions? – Mo Beigi Feb 14 '14 at 09:59
  • i had a similar application before. You just need to do the template matching between each frame of video/webcam and the template image. – skm Feb 14 '14 at 10:01
  • After performing template matching between each incoming frame from webcam/video and template image, you will get a resulting image based upon which you can decide whether the template is present in the given webcam/video frame or not. – skm Feb 14 '14 at 10:08
  • Ah do you have any code you could possibly share with me? Not quite sure how to monitor the incoming frames from the screen. – Mo Beigi Feb 14 '14 at 10:59
  • sorry, i cannot share the code but i can suggest you the method...will update a new answer in some time. – skm Feb 14 '14 at 11:05
1
void start()
{

    VideoCapture cap(0);
    Mat image;
    namedWindow(wndname,1);

    for(;;)
    {
        Mat frame;          
        cap >> frame; // get a new frame from camera

        "Load your template image here"
        "Declare a result image here"
        "Perform the templateMatching() between template image and frame and store the results of correlation in result image declared above"



        char c = cvWaitKey(33);
        if( c == 27 ) break;


   }

}
skm
  • 5,015
  • 8
  • 43
  • 104
  • Thanks for the code but I wish to use the screen (ie desktop frames) instead of video or a camera. – Mo Beigi Feb 15 '14 at 05:29
  • Edit: I've managed to display the desktop in a window using this: http://privatepaste.com/fb2beac94f However there is a type mismatch between frame and templ and hence matchTemplate() fails an assertion. Any idea how to fix this? I got the hwnd2mat function from: http://stackoverflow.com/questions/14148758/how-to-capture-the-desktop-in-opencv-ie-turn-a-bitmap-into-a-mat Many thanks! – Mo Beigi Feb 15 '14 at 06:34
  • it might be because you didn't declare the size of resulting image (matrix which will store the result of MatchTemplate ()) properly. Have a look at `/// Create the result matrix` `int result_cols = img.cols - templ.cols + 1;` `int result_rows = img.rows - templ.rows + 1;` `result.create( result_cols, result_rows, CV_32FC1 );` – skm Feb 15 '14 at 08:03
  • I did try using that exact declaration with no luck. I'll try to fiddle around with it some more, if I can get this working then I can most likely finish! – Mo Beigi Feb 15 '14 at 13:22