Are there any libaries to compare a small image to a screen grab and return an x,y coordinate list of where it's located?

Question

this is a pretty rough area (fuzzy logic image comparison) that is way above my pay grade, so to speak.

I'm trying to write a program that will detect carrots (and other veggies) onscreen from a game and return matching x,y coordinates for each found match.

The intent is to track each time the vegetable grows (5 times) and eventually automate the watering/harvesting of the vegetables (once you see the screenshots this will make more sense).

I tried using the image compare code located at http://sites.google.com/site/serinolorenzo/blackmask

it actually works flawlessly when searching this screenshot (and desktop grabs) http://imageshack.us/photo/my-images/24/mac8.png/

for instances this image... http://imageshack.us/photo/my-images/249/carrotplant.png/

It does find and return multiple "Carrot Seeds" menu coordinates.

However when I get down to detecting the veggies (the meat and potatoes of what the compare needs to detect) it fails to find a match.

I've tried various "carrot" screenshot, most recently... http://imageshack.us/photo/my-images/829/carrotlight1t.png/

to the image http://imageshack.us/photo/my-images/408/macnolight1.png/

(I had to turn in game lighting off when I got here, to ensure colors stayed the same regardless of game time).

I thought (by going over the source code for the image compare) that black would be ignored, so I'm not sure exactly what's going on, but it just will not find a match now, even if I set the thresholds lower.

Are there any (well documented) libraries that exist to do this, as coding it from scratch is just something I can't do at the moment.

Barring that (and if whomever is reading this has the time) can you see where I'm going wrong?

The code I'm using to do the testing is...

    public void tester()
    {
        Bitmap screenshot;
        screenshot = new Bitmap(@"c:\MyMercurial\ATITDVeggieMacro\ATITDVeggieMacro\Images\macnolight1.bmp");

        Bitmap searchFor;
        searchFor = new Bitmap(@"c:\MyMercurial\ATITDVeggieMacro\ATITDVeggieMacro\Images\wholecarrot.bmp");
        int ac1;
        int ac2;

        ArrayList test = BlackMask.Search(screenshot,searchFor,
            0,10,75,5,80,20,100,true,out ac1,out ac2);

        statusUpdate = ac1.ToString()+"good,"+ ac2.ToString()+" bad/ X=";
    }

(I just update the file paths when I want to change images).

(Bonus question, how to I use relative paths to tell it to just go up two directories and then down into the Images folder, I can't at all remember the relative path shortcuts)

Thanks for any help guys, this ones got me slamming my forehead.

Answer 1

For intelligent image processing in .NET, you could look into using AForge.NET. There's also OpenCV, but then are no .NET bindings that I know of off hand (they appear to have some for Java though...). Once you figure out which library, I'm sure you can find additional articles via Google on their usage (besides their standard examples)

I've never played ATITD, however, if you figured out the various specs of how it was developed, that could shed some light into how you might approach this problem in an entirely different direction. For example, if their Client app links to the MS C runtime DLL, it might have been developed in C/C++ (or maybe with some other lang whose framework runs on a C/C++ core). You could end up just manipulating the memory of the program if you found out how it stores the copy of the game state in the Client memory (assuming it's just not some video feed, a la OnLive). If you figure out which rendering framework they use, OpenGL, DX, etc, you may be able to do the image capture more efficiently (and possibly without as much artifacts, like the grass and such, that you have to mask out yourself) via DLL hooks/proxies.