38

What's the way OpenCV handles transparency in image during template matching?

The problem is that the template image needs to have transparent parts, because in the original image there could be anything at those places.

I tried all of the methods, and none of them yielded positive results (e.g. position of template in original image wasn't detected correctly).

Stephen Kennedy
  • 20,585
  • 22
  • 95
  • 108
mnn
  • 1,952
  • 4
  • 28
  • 51

8 Answers8

23

It doesn't seem like OpenCV handles alpha the way you want it to.

You have two options:

  1. Write your own cross-correlation method that will use the alpha channel
  2. Transform your images so your alpha channel becomes irrelevant

Since the first option is straightforward, I will explore the second option here. I'm going to re-use the sample code I provided to a similar question earlier. If you apply cross-correlation directly to your images, the background interferes with the template matching (in particular, light background parts). If you play around with color channels, you will find that matching in the blue channel gives the correct result. This depends on the image content and isn't a consistent way to solve the problem.

Another option is to perform edge detection (e.g. Sobel) on the image and template, and perform cross-correlation then. Here are the edge detected images (I used the Sobel edge detector on the Luma channel in GIMP, and then some intensity stretching).

map

building

As you can see, the alpha channel here has become irrelevant, as most of the terrain has become zero intensity and will not contribute to the cross-correlation calculation. So now cross-correlation can be directly applied, giving the desired result:

misha@misha-desktop:~/Desktop/stackoverflow$ python cross-correlation.py map-blue.png building-maskz-blue.png 
(163, 244)

Finally, here's another related question.

PS. What game is this?

Community
  • 1
  • 1
mpenkov
  • 21,621
  • 10
  • 84
  • 126
  • Thanks, but I didn't get the same results using Sobel on grayscaled image & template (see question). The image is from is an old DOS game - Earth 2140. – mnn Jan 25 '11 at 13:20
  • The reason your images didn't work is because areas where there are no edges are **not black** (they are 127 neutral gray). Compare them to my images. You need the non-edges areas to be **zero** so they don't interfere with the cross-correlation calculation. – mpenkov Jan 25 '11 at 13:44
  • OK, I used cvConvertAbsScale to get non-edges areas to be zero. (see question) However, still my Sobel isn't the same as yours (especially the template). Could it be the fact that I used OpenCV and you used GIMP for Sobel? – mnn Jan 25 '11 at 14:07
  • Most likely -- as I mentioned, I did Sobel followed by some intensity scaling (since the two images were scaled the same way, this didn't affect the auto-correlation, but made things easier to see). Even if the images aren't exactly the same, but as long as the edges are hi and the non-edges are low, the approach will work. Your Sobel image looks like not all the edges (compare the horizontal edges) are being picked up. If it's bothering you, post your code and I'll play around with it tomorrow sometime. – mpenkov Jan 25 '11 at 14:47
  • No, actually Sobel technique is amazing. It always makes positive results, when there's a template located in original image (even a bit obscured, as in the example in question). However I'm having trouble with _multiple occurences_ of template within an image (especially, when there isn't template located in an image - see question for examples). – mnn Jan 26 '11 at 00:00
  • Edge detection may be a workaround in some specific cases. But what if the template is a rainbow with sliding colors? There are no edges. Your trick may work sometimes but it is an ugly workaround for a function that is missing in OpenCV: a real masked template matching. – Elmue Sep 29 '14 at 04:04
  • I think it's age of empires @mpenkov – Dan Jan 01 '23 at 01:24
16

OpenCV 3.0 offers native support for template matching with masked templates. Refer to the new documentation:

Parameters:

image ...

templ ...

result ...

method ...

mask Mask of searched template. It must have the same datatype and size with templ. It is not set by default.

[Slight Digression]

Note that template matching with masked reference images (the larger image) is not possible though. And that makes sense, given OpenCV uses FFT based template matching.

Therefore, if you need to perform template matching only at specific regions of your reference images, you will need to implement your own method for that or mask the output of cv::matchTemplate.

Implementing it from scratch should compensate for cases where you only want to search for the template at very specific regions (i.e.: around harris corners).

Marcello Salomao
  • 161
  • 1
  • 3
15

I have a slightly more brain-dead solution to this issue which actually seems to work reasonably well: Replace the template image's alpha channel with noise, which more or less makes the transparent regions statistically insignificant during the matching process.

For example, my use case involved searching for emoji characters in screen captures from iOS. The iOS keyboard background changes colors depending on context, which makes the matching process problematic if you commit to a particular background color in your template image.

Here's the raw template image on alpha:
raw template image on alpha

Here's the processed template with noise filling in for the alpha channel:
enter image description here

I sent the processed template image through the Template Matching sample code provided in the OpenCV documentation. On either dark or light background, the match is found with reasonable confidence.

Searching on dark backgrounds:

matched on dark

Searching on light backgrounds:

matched on light

In comparison, leaving the template's alpha channel transparent — or committing to a dark or light background — did not return acceptable matches.

kitschpatrol
  • 454
  • 4
  • 6
  • 5
    Your solution is a badly functioning workaround. While normally matchTemplate() returns match certainties of 99% or even 100% if the images are identical, your solution with your sample images returns a certainty of 23% for the matching image. The next not-matching image (one of the smilies) is at 11%. This is a very bad distance between matching (23%) and non-matching (11%) images. And a smiley is COMPLETELY another thing than your template image. So this workaround depends strongly on the threshold that you use to distinguish between match and non-match. Your solution gives very weak results – Elmue Sep 29 '14 at 04:24
  • 2
    So a better workaround would be the following: Use your method (noised template) in a first step to find the possible locations of matches with matchTemplate() and then in a second step mask the transparent parts to black in both: the template and the main image at the locations found in the first step to get the real certainties (up to 100%). – Elmue Sep 29 '14 at 04:41
  • 1
    You would get better certainty results if you would remove the useless noise around the emoji. You have above and below 16 pixels of noise and 5 pixels on each side. After removing them the certainty increases from 23% to 57%. The more noise the worse the recognition! – Elmue Sep 29 '14 at 19:49
1

The SQDIFF/SQDIFF_N option would be a solution if you tried to replace alpha channel with the black RGB color. At least this was my solution to same problem. From my result is obvious that this method is sensitive to brighter pixel values, and I took an opportunity of that.

eboix
  • 5,113
  • 1
  • 27
  • 38
Marstep
  • 11
  • 1
1

OpenCV handles transparency as being part of the image instead of ignoring it, which may cause unintentional results. The way I handle it, is by using a template with transparency as a template and a mask parameter in matchTemplate(). I've answered a similar question here with a bit more detail, maybe it helps.

Zebiano
  • 373
  • 5
  • 13
-1

I think you're trying to do what in OpenCV is called template matching with a mask. I think you could try setting a ROI (region of interest) on the template. This SO question shows how to do it. (note that in that question the ROI is set on the target image, not the template, but the procedure is the same).

Community
  • 1
  • 1
carlosdc
  • 12,022
  • 4
  • 45
  • 62
  • Interesting, but doesn't really help, because I can't narrow the search down to such region (the template image could be located anywhere on the original image). – mnn Jan 24 '11 at 19:51
  • Right. But but in the template itself there are pixels that are transparent (ie that should not be in the template ROI) and pixels that are not transparent (ie that should be in the template ROI). The worst thing that can happen (as @Utkarsh Shinha says is that you have to have to write your own template matching function to ignore the pixels not in the ROI). – carlosdc Jan 25 '11 at 00:38
  • Look at example images. The transparent region is not rectangle in any way, and ROI is a rectangle. – mnn Jan 25 '11 at 10:05
  • carlosdc, you did not understand the question. – Elmue Sep 27 '14 at 02:33
-1

I'm not sure, but the transparency channel is treated just like any other channel. If a pixel in a template is "transparent" it should be "transparent" on the main image as well. I'm just guessing here.

Utkarsh Sinha
  • 3,295
  • 4
  • 30
  • 46
  • And that's the problem. I need template matching to _ignore_ pixels with transparency in template image. Otherwise I'll never be able to find template in original image, because in the original image, there could be anything around the object I'm looking for – mnn Jan 24 '11 at 19:50
  • Get rid of the transparency channel in both. That could work. Or you could write your own template matching function. The OpenCV documentation lists the formulae use for the various methods. You can modify them so they "respect" the amount of transparency at a pixel. – Utkarsh Sinha Jan 24 '11 at 20:44
  • Utkarash, you are right: You are just guessing. But the real world is much more complicated than you might guess. When you write your own matching function comparing the image with the template pixel by pixel using the given formulae this will be unacceptable slow (running up to a minute) even in speed optimized C++. The reason that OpenCV is so extremely fast is that it uses DFT (Fourier transform) in matchTemplate(). But the code is so extremely complicated (and free of any comments) that only a mathematician will understand it. So forget writing your own matchTemplate() function! – Elmue Sep 29 '14 at 03:14
-2

I came across the same problem and I thought of a solution. Asuming that referenceImageMask and templateMask have 1s in the good pixels and 0s in the bad ones. And that referenceImage and templateImage have already been masked and have 0s in the bad pixels as well.

Then, the first result of template matching will give the not normalized cross correlation between the images. However, a bunch of pixels were zero.

The second template matching will give for each possible offset the number of pixels that were at the same time different from zero (unmasked) in both images.

Then, normalizing the correlation by that number should give the value you (and I) wanted. The average product for the pixels that are not masked in both images.

Image<Gray, float> imCorr = referenceImage.MatchTemplate(templateImage,      Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCORR);
Image<Gray, float> imCorrMask = referenceImageMask.MatchTemplate(templateMask, Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCORR);
_imCorr = _imCorr.Mul(_imCorrMask.Pow(-1));

UPDATE: actually, this solution does not work. Because the implementation of the cross correlation in opencv uses the DFT there will be numeric issues and you cannot use the second crosscorrelation to correct the first one.

Jorge Otero-Millan
  • 49
  • 3
  • 10