0

I am wondering how I could search a set rectangle on the screen and have it compare to an image that I specify to see if it matches?

Lets say it could Search x1 y1 to x2 y2 and compare against an image? and return the boolean?

I know Auto-it has a similar function seen here: http://www.autohotkey.com/docs/commands/ImageSearch.htm

Has anyone done this that they could reference? I am using vb.net.

EDIT: Abdias, I have put your code into a class instead and I am calling it like this:

     Dim bm As Bitmap = Bitmap.FromFile(Label1.Text)
    Dim bm2 As Bitmap = Bitmap.FromFile(Label2.Text)
    Dim pnt As Point = ImageFinder.Contains(bm, bm2)
    If pnt <> Nothing Then
        MessageBox.Show("Possible match found at " & pnt.X.ToString() & " " & pnt.Y.ToString())
    Else
        MessageBox.Show("No match.")
    End If

It seems that every set of images I try return no point. Even though they 100% contain eachother. I took an image and cropped it by a couple px and still did not return a match. I have made sure the source is larger. I tried saving a couple images as 24 bit jpg in paint and still nothing.

Here are two sample images.enter image description hereenter image description here

user1632018
  • 2,485
  • 10
  • 52
  • 87
  • The jpeg format will change a lot of pixels/colors and even if the images appear to be the same the won't be (at pixel level). My routine is pixel based which means it need to match the exact pixel and color (color-tolerance can easily be implemented though). Try to save as PNG or BMP instead - these are loss-less. The option is to use a very different approach utilizing feature recognition and so forth. That was outside the scope of my little routine and is a more complex area. –  Dec 31 '12 at 20:20
  • Two dimensional cross-correlation may work with a threshold set for what is considered a match. – Alan Jan 02 '13 at 17:06

2 Answers2

1

I made this function which can see if an image exist within a bigger image. It is written as an extension, but can easily be modified to a normal function as well as supporting a region.

To use it:

  • Load main image into standard Bitmap as bmp
  • Load image to look for into bmpSearch

Then call:

Dim pt as Point = bmp.Contains(bmpSearch)
If pt <> Nothing Then
    '... image found at pt
End If

The code for the extension (room for optimizations, but written as a 20 minute exercise for another question on this site):

'
'-- Extension for Bitmap
'
<Extension()>
Public Function Contains(src As Bitmap, ByRef bmp As Bitmap) As Point
    '
    '-- Some logic pre-checks
    '
    If src Is Nothing OrElse bmp Is Nothing Then Return Nothing

    If src.Width = bmp.Width AndAlso src.Height = bmp.Height Then
        If src.GetPixel(0, 0) = bmp.GetPixel(0, 0) Then
            Return New Point(0, 0)
        Else
            Return Nothing
        End If
    ElseIf src.Width < bmp.Width OrElse src.Height < bmp.Height Then
        Return Nothing
    End If
    '
    '-- Prepare optimizations
    '
    Dim sr As New Rectangle(0, 0, src.Width, src.Height)
    Dim br As New Rectangle(0, 0, bmp.Width, bmp.Height)

    Dim srcLock As BitmapData = src.LockBits(sr, Imaging.ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb)
    Dim bmpLock As BitmapData = bmp.LockBits(br, Imaging.ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb)

    Dim sStride As Integer = srcLock.Stride
    Dim bStride As Integer = bmpLock.Stride

    Dim srcSz As Integer = sStride * src.Height
    Dim bmpSz As Integer = bStride * bmp.Height

    Dim srcBuff(srcSz) As Byte
    Dim bmpBuff(bmpSz) As Byte

    Marshal.Copy(srcLock.Scan0, srcBuff, 0, srcSz)
    Marshal.Copy(bmpLock.Scan0, bmpBuff, 0, bmpSz)

    ' we don't need to lock the image anymore as we have a local copy
    bmp.UnlockBits(bmpLock)
    src.UnlockBits(srcLock)

    Dim x, y, x2, y2, sx, sy, bx, by, sw, sh, bw, bh As Integer
    Dim r, g, b As Byte

    Dim p As Point = Nothing

    bw = bmp.Width
    bh = bmp.Height

    sw = src.Width - bw      ' limit scan to only what we need. the extra corner
    sh = src.Height - bh     ' point we need is taken care of in the loop itself.

    bx = 0 : by = 0
    '
    '-- Scan source for bitmap
    '
    For y = 0 To sh
        sy = y * sStride
        For x = 0 To sw

            sx = sy + x * 3
            '
            '-- Find start point/pixel
            '
            r = srcBuff(sx + 2)
            g = srcBuff(sx + 1)
            b = srcBuff(sx)

            If r = bmpBuff(2) AndAlso g = bmpBuff(1) AndAlso b = bmpBuff(0) Then
                p = New Point(x, y)
                '
                '-- We have a pixel match, check the region
                '
                For y2 = 0 To bh - 1
                    by = y2 * bStride
                    For x2 = 0 To bw - 1
                        bx = by + x2 * 3

                        sy = (y + y2) * sStride
                        sx = sy + (x + x2) * 3

                        r = srcBuff(sx + 2)
                        g = srcBuff(sx + 1)
                        b = srcBuff(sx)

                        If Not (r = bmpBuff(bx + 2) AndAlso
                                g = bmpBuff(bx + 1) AndAlso
                                b = bmpBuff(bx)) Then
                            '
                            '-- Not matching, continue checking
                            '
                            p = Nothing
                            sy = y * sStride
                            Exit For
                        End If

                    Next
                    If p = Nothing Then Exit For
                Next
            End If 'end of region check

            If p <> Nothing Then Exit For
        Next
        If p <> Nothing Then Exit For
    Next

    bmpBuff = Nothing
    srcBuff = Nothing

    Return p

End Function
  • 1
    You might menton that you hardcoded the pixel format to RGB24. Have you tried ARGB32? That would allow to use a single comparison of Int32. – igrimpe Dec 14 '12 at 08:10
  • Thats btw one of the few cases, where c# would have a definitive advantage, because you can work in an unsafe context and use pointers directly. And afaik a single scanline (at least on x64) is always 8 byte aligned and has a length of a multiple of 8. Then you can even work with (on x64 again) "native" 64 bit values, which might speed it up even more. – igrimpe Dec 14 '12 at 08:27
  • Yes, I miss that in vb (I'm not a "c# guy" though, was an "assembler guy"). Yeah, the memory is always aligned on 4-bytes (or 8-bytes in a 64-bit env). –  Dec 14 '12 at 08:30
  • Thank you Abdias, I am testing your and igrimpes solutions as we speak. – user1632018 Dec 14 '12 at 22:27
  • Hi Abdias, thank you for comming back. I did not get the chance to fully test your solution out, when I came back on here your post was gone. I edited my post, it is mainly directed to you I am hoping you can tell me where I am going wrong. I basically took the code as is is except removed the extension code and added it into a class. – user1632018 Dec 31 '12 at 08:40
0

I've played around a bit and thought that using something like the KMP algorithm (that's an easy one to implement I guess -> wikipedia has a nice pseudocode) might be helpfull too:

Imports System.Drawing.Imaging
Imports System.Runtime.InteropServices

Public Class ImageFinder

Public Shared Function Contains(Parent As Bitmap, Child As Bitmap) As Point

    If Parent Is Nothing OrElse Child Is Nothing Then Throw New ArgumentException("Narf!")
    If Parent.PixelFormat <> Imaging.PixelFormat.Format32bppArgb OrElse Child.PixelFormat <> Imaging.PixelFormat.Format32bppArgb Then Throw New ArgumentException("Narf again!")

    If Parent.Width = Child.Width AndAlso Parent.Height = Child.Height AndAlso Parent.GetPixel(0, 0) <> Child.GetPixel(0, 0) Then Return Nothing
    If Child.Width > Parent.Width OrElse Child.Height > Parent.Height Then Return Nothing

    Dim bmdParent, bmdChild As BitmapData
    Try
        ' Get bitmap data into array of int
        bmdParent = Parent.LockBits(New Rectangle(0, 0, Parent.Width, Parent.Height), ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb)
        bmdChild = Child.LockBits(New Rectangle(0, 0, Child.Width, Child.Height), ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb)

        Dim ParentValuesPerLine As Integer = bmdParent.Stride \ 4
        Dim ChildValuesPerLine As Integer = bmdChild.Stride \ 4

        Dim ParentData((bmdParent.Stride \ 4) * bmdParent.Height - 1) As Integer
        Dim ChildData((bmdChild.Stride \ 4) * bmdChild.Height - 1) As Integer

        Marshal.Copy(bmdParent.Scan0, ParentData, 0, ParentData.Length)
        Marshal.Copy(bmdChild.Scan0, ChildData, 0, ChildData.Length)

        If bmdParent IsNot Nothing Then Parent.UnlockBits(bmdParent)
        bmdParent = Nothing
        If bmdChild IsNot Nothing Then Child.UnlockBits(bmdChild)
        bmdChild = Nothing

        ' Create KMP-Table:
        Dim T(Child.Height - 1)() As Integer
        For i = 0 To Child.Height - 1
            T(i) = KMP_Table(ChildData, i * ChildValuesPerLine, ChildValuesPerLine)
        Next

        Dim line_c As Integer = 0
        Dim line_p As Integer = 0
        Dim found As Boolean

        While line_p <= Parent.Height - Child.Height
            line_c = 0
            Dim childoffset As Integer = line_c * ChildValuesPerLine
            Dim parentoffset As Integer = line_p * ParentValuesPerLine


            Dim m As Integer = -1
            While True
                m = KMP_Search(ParentData, parentoffset, ParentValuesPerLine, m + 1, ChildData, 0, ChildValuesPerLine, T(0))
                If m > -1 Then
                    ' first line found
                    Debug.Print("Possible match at {0},{1}", m, line_p)
                    found = True
                    Dim p = parentoffset + ParentValuesPerLine
                    Dim c = childoffset + ChildValuesPerLine
                    For i = 1 To Child.Height - 1
                        If KMP_Search(ParentData, p, ParentValuesPerLine, m, ChildData, childoffset, ChildValuesPerLine, T(i)) <> m Then
                            ' this line doesnt match
                            found = False
                            Exit For
                        End If
                        p += ParentValuesPerLine
                        c += ChildValuesPerLine
                    Next
                    If found Then
                        Debug.Print("Found match at {0},{1}", m, line_p)
                        Return New Point(m, line_p)
                    End If
                Else
                    Exit While
                End If
            End While

            line_p += 1
        End While

        'Catch ex As Exception
        'Throw
    Finally
        If bmdParent IsNot Nothing Then Parent.UnlockBits(bmdParent)
        If bmdChild IsNot Nothing Then Child.UnlockBits(bmdChild)
    End Try

End Function

Private Shared Function KMP_Search(ByVal S As Integer(), s0 As Integer, slen As Integer, m As Integer,
                                   ByVal W As Integer(), w0 As Integer, wlen As Integer,
                                   tbl() As Integer) As Integer

    Dim i As Integer = 0

    While m + i < slen
        If W(w0 + i) = S(s0 + m + i) Then
            If i = wlen - 1 Then Return m
            i += 1
        Else
            m = m + i - tbl(i)
            If tbl(i) > -1 Then
                i = tbl(i)
            Else
                i = 0
            End If
        End If

    End While

    Return -1


End Function

Private Shared Function KMP_Table(ByRef arr() As Integer, start As Integer, count As Integer) As Integer()

    Dim table(count - 1) As Integer
    table(0) = -1
    table(1) = 0

    Dim pos As Integer = 2
    Dim cnd As Integer = 0

    While pos < count - 1
        If arr(start + pos - 1) = arr(start + cnd) Then
            cnd += 1
            table(pos) = cnd
            pos += 1
        ElseIf cnd > 0 Then
            cnd = table(cnd)
        Else
            table(pos) = 0
            pos += 1
        End If
    End While

    Return table

End Function

End Class

I used the following "benchmark" function to compare against the code from Abdias Software :

Private Sub Button1_Click_1(sender As Object, e As EventArgs) Handles Button1.Click

    Dim ofd As New OpenFileDialog
    If ofd.ShowDialog = Windows.Forms.DialogResult.OK Then
        Dim bm As Bitmap = Bitmap.FromStream(New MemoryStream(File.ReadAllBytes(ofd.FileName)))
        Dim bm2 As New Bitmap(bm.Width, bm.Height, PixelFormat.Format32bppArgb)
        Dim gr = Graphics.FromImage(bm2)
        gr.DrawImageUnscaled(bm, New Point(0, 0))
        Dim bm3 As New Bitmap(100, 100, PixelFormat.Format32bppArgb)
        gr = Graphics.FromImage(bm3)
        gr.DrawImage(bm2, New Rectangle(0, 0, 100, 100), New Rectangle(bm2.Width - 110, bm2.Height - 110, 100, 100), GraphicsUnit.Pixel)
        PictureBox1.Image = bm3
        Dim res As New List(Of Integer)
        For i = 1 To 10
            Dim stp = Stopwatch.StartNew
            Dim k = ImageFinder.Contains(bm2, bm3)
            stp.Stop()
            res.Add(stp.ElapsedMilliseconds)
        Next
        ListBox1.Items.Add(String.Format("KMP: Image = {3}x{4}, Min = {0}, Max = {1}, Avg = {2}", res.Min, res.Max, res.Average, bm2.Width, bm2.Height))
        res.Clear()
        For i = 1 To 10
            Dim stp = Stopwatch.StartNew
            Dim k = bm2.ContainsSO(bm3)
            stp.Stop()
            res.Add(stp.ElapsedMilliseconds)
        Next
        ListBox1.Items.Add(String.Format("SO: Image = {3}x{4}, Min = {0}, Max = {1}, Avg = {2}", res.Min, res.Max, res.Average, bm2.Width, bm2.Height))
    End If

End Sub

I tested with large (8MP) and small (1MP) photos (no drawings, icons, etc) and found the kmp version roughly 2 times faster than the other approach. In absoulute numbers 40ms vs 75ms for an 8MP image tested on i7-2600. The result might depend on the types of images, because KMP (and others) win, when they can skip larger areas.

igrimpe
  • 1,775
  • 11
  • 12
  • Im going to try this out right now. I will report back. Thanks for taking the time to measure the speed. – user1632018 Dec 14 '12 at 22:26
  • Im trying to figure out how to use a string(image path) for the second parameter of the contains function but can't figure it out. – user1632018 Dec 15 '12 at 01:08
  • Do I have to open it to the memory from the path first then input the name into the second paramter? – user1632018 Dec 15 '12 at 01:09
  • Nevermind I figured it out. Dim bm As New Bitmap("image path") – user1632018 Dec 15 '12 at 01:13
  • After testing this out fully, I found that it doesn't work all too well. The image has to be identical. Even when I took an image and cropped it substantially it still did not work. – user1632018 Dec 31 '12 at 07:14
  • You didn't specify, that you are looking for some kind of "fuzzy" matching. If you for example look for an algo that finds out if "x is 5", you can not say, the algo doesn't work, if it doesn't match "5.1". – igrimpe Jan 02 '13 at 05:58