10

I am trying to understand how I can use PIL in Python 2.7 to search the whole screen for a certain image and click on it. I've been searching around and haven't been able to find a solution. I want to create a small GUI with one button in the middle of it that when clicked will search the entire screen for a predefined image. Once the image is found the program will then click in the centre of it and end. In short the program will detect if an image is present on the users screen and click it.

I did find an interesting bit on Sikuli, but that doesn't help me because it's unable to export to an .exe.

The image that the program will look for will most likely be in the same place each time it searches, but I didn't want to hard-code the location as it has the potential to move and I don't want that being an issue later on.

What I need is the code method I would use to search for the image on screen and send back the cords to a variable.

Image explanation/example: Image

Reference image of rifle: Image2

Mystra007
  • 275
  • 1
  • 9
Freddie
  • 103
  • 1
  • 1
  • 7
  • so the image on the screen will be of same size or different as reference image ? – Sainath Motlakunta Dec 07 '14 at 15:12
  • Yes it will be different. Roughly around the same size. – Freddie Dec 07 '14 at 15:29
  • Searching for an exact image is easy. Searching for a similar image requires an image recognition algorithm, and is substantially harder. – Gabe Dec 07 '14 at 15:48
  • No it will be the exact same image every time. I showed the rifle because I haven't made the other one yet. – Freddie Dec 07 '14 at 15:53
  • even if it is a same image if it is of different size it is would be difficult task and may require more complicated algorithms – Sainath Motlakunta Dec 07 '14 at 16:44
  • Here is what the button should look like http://i.imgur.com/y8X7shb.png . The image is a constant and will not change its size or look, only the position on screen. – Freddie Dec 07 '14 at 16:50

3 Answers3

11

PIL is the wrong tool for this job. Instead you should look into openCV (open source computer vision), which has fantastic python bindings. Here is a link to an example (in C but should be easy to redo with the python bindings) that does what you are looking for, but even allows the image to be rotated, scaled, etc.

http://docs.opencv.org/doc/tutorials/features2d/feature_homography/feature_homography.html http://docs.opencv.org/doc/tutorials/features2d/detection_of_planar_objects/detection_of_planar_objects.html

Edit:

I assume you are using windows, as your example image looks like window. In this case you can use:

from PIL import ImageGrab
pil_img = ImageGrab.grab()
opencv_img = numpy.array(pil_img)

then use opencv to process the image to find sub image you are looking for.

If you want to do this cross platform, then you will need to use wxWidgets to do the screengrab: https://stackoverflow.com/a/10089645/455532

Community
  • 1
  • 1
Kyle
  • 663
  • 6
  • 6
3

Even I wanted to do the same but using different module - pyautogui. I finally found the solution for my problem and I am sure this solution will also help you. You have to just go to this webpage and read the locate function topic completely and you'll be able to solve your problem.

loved.by.Jesus
  • 2,266
  • 28
  • 34
I Answer StackOvrFlw questions
  • 39
  • 4
1

I recommend you give a look on PyAutoGUI, a well documented library to control mouse and keyboard, also can locate imagens on screen, find the position, move the mouse to any location and clicks on location, also can simulate drag and drop, type on input fields, give double clicks and much more.

Samuel Santos
  • 85
  • 5