Matching images from file with Sikuli

Question

I just found about Sikuli when I was looking for a library to find matches of a given image within a larger image (both loaded from files). By default, Sikuli only supports loading the searched image from file, but relies on a proprietary class Screen to take screenshots to use as base for the search... And I'd like to have the ability to use a image file instead.

Looking for a solution has led me to this question, but the answer is a bit vague when you consider that I have no prior experience with Sikuli and the available documentation is not particularly helpful for my needs.

Does anyone have any examples on how to make a customized implementation of Screen, ScreenRegion, ImageScreen and ImageScreenLocation? Even a link to a more detailed documentation on these classes would be a big help.

All I want is to obtain the coordinates of an image match within another image file, so if there's another library that could help with this task I'd more than happy to learn about it!

Answer 1

You can implement it by yourself with something like this:

class MyImage{
    private BufferedImage img;
    private int imgWidth;
    private int imgHeight;

    public MyImage(String imagePath){       
        try{
            img = ImageIO.read(getClass().getResource(imagePath));
        }catch(IOException ioe){System.out.println("Unable to open file");}
        init();
    }

    public MyImage(BufferedImage img){
        this.img = img;
        init();
    }

    private void init(){
        imgWidth = img.getWidth;
        imgHeight = img.getHeight();
    }

    public boolean equals(BufferedImage img){
        //Your algorithm for image comparison (See below desc for your choices)
    }

    public boolean contains(BufferedImage subImage){
        int subWidth = subImage.getWidth();
        int subHeight = subImage.getHeight();
        if(subWidth > imgWidth || subHeight > imgHeight)
            throw new IllegalArgumentException("SubImage is larger than main image");

        for(int x=0; x<(imgHeight-subHeight); x++)
            for(int y=0; y<(imgWidth-subWidth); y++){
                BufferedImage cmpImage = img.getSumbimage(x, y, subWidth, subHeight);
                if(subImage.equals(cmpImage))
                    return true;
            }
        return false;
    }
}

The contains method will grab a subimage from the main image and compare with the given subimage. If it is not the same, it will move on to the next pixel until it went through the entire image. There might be other more efficient ways than moving pixel by pixel, but this should work.

To compare 2 images for similarity

You have at least 2 options:

1. Scan pixel by pixel using a pair of nested loop to compare the RGB value of each pixel. (Just like how you compare two int 2D array for similarity)

2. It should be possible to generate a hash for the 2 images and just compare the hash value.

Answer 2

Aah... Sikuli has an answer for this too... You just didnt look close enough. :) Answer : The FINDER Class

Pattern searchImage = new Pattern("abc.png").similar((float)0.9);
String ScreenImage = "xyz.png"; //In this case, the image you want to search
Finder objFinder = null;
Match objMatch = null;
objFinder = new Finder(ScreenImage);
objFinder.find(searchImage); //searchImage is the image you want to search within ScreenImage
int counter = 0;
while(objFinder.hasNext())
{
    objMatch = objFinder.next(); //objMatch gives you the matching region.
    counter++;
}
if(counter!=0)
System.out.println("Match Found!");

Answer 3

In the end I gave up on Sikuli and used pure OpenCV in my Android project: The Imgproc.matchTemplate() method did the trick, giving me a matrix of all pixels with "scores" for the likehood of that being the starting point of my subimage.

Answer 4

With Sikuli, you can check for the presence of an image inside another one. In this example code, the pictures are loaded from files. This code tell us if the second picture is a part of the first picture.

public static void main(String[] argv){
    String img1Path = "/test/img1.png";
    String img2Path = "/test/img2.png";
    if ( findPictureRegion(img1Path, img2Path) == null )
        System.out.println("Picture 2 was not found in picture 1");
    else
        System.out.println("Picture 2 is in picture 1");
}

public static ScreenRegion findPictureRegion(String refPictureName, String targetPictureName2){
    Target target = new ImageTarget(new File(targetPictureName2));
    target.setMinScore(0.5); // Precision of recognization from 0 to 1.
    BufferedImage refPicture = loadPicture(refPictureName);
    ScreenRegion screenRegion = new StaticImageScreenRegion(refPicture);
    return screenRegion.find(target);
}

public static BufferedImage loadPicture(String pictureFullPath){
    try {
        return ImageIO.read(new File(pictureFullPath));
    } catch (IOException e) {
        e.printStackTrace();
        return null;
    }
}

To use Sikuli package, I added this dependency with Maven :

    <!-- SIKULI libraries -->
    <dependency>
        <groupId>org.sikuli</groupId>
        <artifactId>sikuli-api</artifactId>
        <version>1.1.0</version>
    </dependency>