0

I'm trying to implement a function that will scan the computer screen for a given image of an icon and get its coordinates.

Since the image of the icon will be taken by me I thought it would be best not to use any image recognition technique but rather byte comparison, thinking if I take a screenshot of the screen, while the icon is visible, I will cut just the icon, save it as *.bmp and then instruct the program to take a Bitmap screenshot, convert it and the icon image to a byte array and look for one in the other and, perhaps, by the location of the bytes of the icon image in the screenshot determine where on the screen it is.

I fail at step 1, which is - use the "Prt Scr" button to take a screenshot, save it as *.bmp then copy the part with the icon (in this case The Start Button), save only that as *.bmp separately and then use the ImageConverter class to convert them both into byte arrays and look for one in the other.

Initially I've tried most all the answers from this question but they all failed to find it. Then I got a StringBuilder and iterated through each array, appending each element to the StringBuilder with a comma after it. Then I copied both results to Notepad++ and searched for the Start Button bytes in the full screenshot. They aren't there.

So I wonder why the bytes wouldn't be there, considering one is a direct copy of a smaller section of the bigger picture and, two - is my approach to find the coordinates of a smaller image inside a larger one by using byte comparison wrong?

EDIT: Removed code as it doesn't matter in this case.

user14092802
  • 105
  • 10
  • 2
    That is way too much code to wade through. But yes, a bytewise comparison of images may fail for several reasons like compression, dithering, antialiasing or precomputed colors. So I suggest to manually look into examples to find the exact nature of the differences.. – TaW Oct 23 '20 at 13:24
  • @TaW, I thought saving in bmp from Paint would dodge all those problems? – user14092802 Oct 23 '20 at 14:36
  • 1
    Saving an uncompressed format like (some, not all) bmp or (recommended) png is definitely a good idea. I'm not 100% sure about the screen pixels, though. Maybe someone else will weigh in.. – TaW Oct 23 '20 at 15:15
  • Isn't it much simpler to use `LockBits` and just access the actual back-end bytes of the image for comparison? Then you have exactly and only the bytes you want, in the exact format you want, with no headers and all that. But it won't get around the issue that the image may be alpha-blended into something in the desktop environment. Take a desktop icon with smooth edge fades, for example; its pixels will mix into the background pixels. – Nyerguds Oct 24 '20 at 09:21
  • @TaW since when is png uncompressed? – Nyerguds Oct 24 '20 at 09:22
  • Whoops, I meant __lossless__ , of course. – TaW Oct 24 '20 at 09:30
  • @Nyerguds, what is the difference between `LockBits` and `(byte[])converter.ConvertTo(image, typeof(byte[]))` where "converter" is an `ImageConverter`? – user14092802 Oct 26 '20 at 08:48
  • @TaW, what is the difference between an uncompressed and a lossless format? I thought 24bit bmp is the purest thing you can expect as it doesn't compress and saves the pixels exactly as they are, while I expected a png to have some compression and, therefore, changes to the pixels. – user14092802 Oct 26 '20 at 08:49
  • No. You can compress losslessly (png) or lossy (jpg). BMP has [many](https://en.wikipedia.org/wiki/BMP_file_format) subformats, including BI_JPEG, so it takes some care to make sure what happens. PNG is a simple and reliable option. Also: There is a time-saving method to prepare images for display called ['premultiplied'](https://docs.microsoft.com/en-us/dotnet/api/system.drawing.imaging.pixelformat?view=dotnet-plat-ext-3.1), including the alpha values right in the rgb channels, which may or may not interfere.. - So I suggest to dig down into the pixel data and compare a few pixels manually. – TaW Oct 26 '20 at 10:28
  • @user14092802 LockBits does not convert to a file format. It gives you the pixel data of the image in raw bytes. – Nyerguds Oct 26 '20 at 22:12
  • @Nyerguds, I thought converting it to a byte array did the same thing? Does it not? Does the code, using an ImageConverter, that I've cited not do that? – user14092802 Oct 28 '20 at 08:16
  • @user14092802 I have no idea. I honestly never used that. But that way of converting will lack necessary side data; you need the image's stride to handle its byte data correctly, and the `BitmapData` object returned by the `LockBits` will give you access to that. – Nyerguds Oct 28 '20 at 10:16
  • @user14092802 I just tested it. As I assumed, if you look at the resulting bytes of the ImageConverter operation, it's just a png file. Not the internal data at all. – Nyerguds Oct 28 '20 at 11:14
  • @Nyerguds, oh, wow - thank you! This has been most enlightening and very useful or my project! – user14092802 Oct 28 '20 at 11:48
  • @user14092802 I gave a bunch of answers here on SO that deal with handling raw image data; stuff like cropping, identifying regions, converting to colour palette, and taking it from / putting it into `Bitmap` objects. There may be code in there that's useful to you. – Nyerguds Oct 28 '20 at 12:41

0 Answers0