How do I find several key pixel points in a picture using Python?

Question

I know that by Opencv-python's matchtemplate function can match picture 2 from picture 1.

import cv2


template = cv2.imread("1.bmp")
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
template = cv2.Canny(template, 50, 200)


edged = cv2.imread("2.bmp")
edged = cv2.cvtColor(edged, cv2.COLOR_BGR2GRAY)
edged = cv2.Canny(edged, 50, 200)


result = cv2.matchTemplate(edged, template, cv2.TM_CCOEFF_NORMED)
(_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

But if I only have pixels like picture 3, how do I match from picture 1?

picture 3: Some pixels in picture two are extracted and the rest is filled in red.

I didn't find a opencv-python or a PIL solution. I think it can be solved by traversing pixels, but that doesn't guarantee performance. So is there a better way?

Possible duplicate of [matchTemplate with non-rectangular templ](https://stackoverflow.com/questions/27993074/matchtemplate-with-non-rectangular-templ). — WhoIsJack, Jun 28 '18 at 16:47
So I have to turn the picture 3 into alpha? I don't want to do this, I hope the algorithm is able to automatically ignore those simple pixels, detect only those pixels that are prominent. @WhoIsJack — Martin Zhu, Jun 28 '18 at 16:59
Unfortunately, "the algorithm" doesn't automagically ignore "simple" pixels, so you will have to provide a mask. As the answer in the linked question suggests, [matchTemplate](https://docs.opencv.org/3.4/df/dfb/group__imgproc__object.html#ga586ebfb0a7fb604b35a23d85391329be) supports the `mask` parameter for this purpose (as of OpenCV 3). — WhoIsJack, Jun 28 '18 at 17:15
OK, Can I generate a mask layer from the code in picture 3? to ignore the red part.@WhoIsJack — Martin Zhu, Jun 28 '18 at 17:57
@MartinZhu try using keypoint feature extraction like ORB, SIFT, etc.. — Jeru Luke, Jun 29 '18 at 05:53

WhoIsJack · Accepted Answer · 2018-06-29T12:09:53.967

1

According to the answer by Neeraj Komuravalli to this question, matchTemplate supports the mask argument as a way of excluding certain pixels in the template from being considered in the matching (docs).

To generate a mask based on the red pixels, a simple solution is to use a Boolean expression to select those pixels that are 0 in blue and green but 255 in red:

mask = (template[:,:,0]==0) & (template[:,:,1]==0) & (template[:,:,2]==255)
mask = ~mask
mask = mask.astype(np.uint8)

Note that the conversion to uint8 is necessary since the mask must be of the same datatype as the template.

Edit: ~mask inverts the mask (0 becomes 1 and vice versa), which is necessary since 0 indicates the pixels to be masked, at least when using the method cv2.TM_CCORR_NORMED.

Whilst this addresses your question in principle, it will not yield a working solution in this case.

This is because of the Canny edge filter that is being applied to the images. Since there is no way of masking the red pixels in the template when applying Canny, the boundaries of the red pixel regions will affect the result of the edge detection and thus change the template to look quite different from the original.

In this example the matching fails as a consequence, returning a position that is completely wrong.

Removing the Canny steps solves this problem... but it also makes the approach a little less robust/precise. In this case, the matching actually seems to end up a few pixels off from the 100% correct match. Unfortunately, I can't think of any means of improving this.

Here's the complete code that works for me (with the aforementioned caveat in terms of precision):

import cv2
import numpy as np

template = cv2.imread("masked_template.png")

mask = (template[:,:,0]==0) & (template[:,:,1]==0) & (template[:,:,2]==255)
mask = mask.astype(np.uint8)

template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
#template = cv2.Canny(template, 50, 200)

edged = cv2.imread("edged.png")
edged = cv2.cvtColor(edged, cv2.COLOR_BGR2GRAY)
#edged = cv2.Canny(edged, 50, 200)

result = cv2.matchTemplate(edged, template, cv2.TM_CCORR_NORMED, mask=mask)
(_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

edited Jun 29 '18 at 12:09

answered Jun 28 '18 at 23:05

WhoIsJack

1,378
2
15
25

Thank you for your answer. I have a question that `(template[:,:,0]==0) & (template[:,:,1]==0) & (template[:,:,2]==255)` will make the red pixels true (uint8 == 1), however, 0 in the mask indicates occlusion. – Martin Zhu Jun 29 '18 at 07:30
Where did you get this information? I wasn't sure and so I just tested both (you can invert the mask by adding `mask = ~mask` before the `astype` line); assuming that `1` occludes gives a decent match, assuming that `0` occludes doesn't. – WhoIsJack Jun 29 '18 at 08:52
Hm... apparently that also depends on the `method` used. In the answer, I am using `cv2.TM_SQDIFF`, which gives a good result with this mask and a poor result with the inverted mask. However, when using `cv2.TM_CCORR_NORMED` it seems to be the other way around. Perhaps you could use some additional test cases to check whether this observation is consistent or spurious? – WhoIsJack Jun 29 '18 at 08:54
I used `TM_CCORR_NORMED`, because it can get a fractional match rate, it is more intuitive. After testing, It seems to indicate 0 in the mask indicates occlusion . . . – Martin Zhu Jun 29 '18 at 10:57
I found a new problem. when using GRAY, the match degree is 0.9, and when use BGR, the match is only 0.4. This means that when using GRAY, the grayscale will make the error a lot more. ...@WhoIsJack – Martin Zhu Jun 29 '18 at 17:08
Converting from BGR to grayscale obviously looses quite a bit of information, so this is expected. If you want to remove the grayscale conversion all you have to do is make sure that the mask's shape is consistent with the BGR images, i.e. you have to repeat the mask for all three colors. You can do this e.g. with `mask = np.expand_dims(mask, -1).repeat(3, axis=-1)`. – WhoIsJack Jun 29 '18 at 20:22
Side note: StackOverflow comments are not intended for extensive discussion. If my answer addresses your question, please continue developing and come back to StackOverflow if you run into new issues for which you cannot find a solution. Please avoid using the comments as a chat to ask for various improvements to your code. Thanks! – WhoIsJack Jun 29 '18 at 20:31

How do I find several key pixel points in a picture using Python?

1 Answers1