Taking fast screenshot Winapi and Opencv

Question

I need to take very fast screenshots of a game for an OpenCV project I am working on. I can use PIL easily for example:

def take_screenshot1(hwnd):
    rect = win32gui.GetWindowRect(hwnd)
    img = ImageGrab.grab(bbox=rect)
    img_np = np.array(img)
    return cv2.cvtColor(img_np, cv2.COLOR_RGB2BGR)

But it takes on average of 0.05 seconds which isn't fast enough for real time capture.

I can use the answer posted here, but that only saves the bitmap to a file. That is over 10 times faster than by using PIL, but I am unsure of any methods within OpenCV to convert it to a bgr/hsv image.

def take_screenshot(hwnd):
    wDC = win32gui.GetWindowDC(hwnd)
    dcObj=win32ui.CreateDCFromHandle(wDC)
    cDC=dcObj.CreateCompatibleDC()
    dataBitMap = win32ui.CreateBitmap()
    dataBitMap.CreateCompatibleBitmap(dcObj, 500, 500)
    cDC.SelectObject(dataBitMap)
    cDC.BitBlt((0, 0), (500, 500), dcObj, (0, 0), win32con.SRCCOPY)

    dataBitMap.SaveBitmapFile(cDC, "foo.png")

    dcObj.DeleteDC()
    cDC.DeleteDC()
    win32gui.ReleaseDC(hwnd, wDC)
    win32gui.DeleteObject(dataBitMap.GetHandle())
    im = cv2.imread("foo.png")
    return cv2.cvtColor(im, cv2.COLOR_RGB2BGR)

EDIT:The size of the window is 500x500, so it is saving the same area in both examples.

Even if I save the image, and then reopen it with OpenCV it is still faster than PIL, but surely there is an easier way?

EDIT: Ok so using the comments and doing some research on winapi I now can access the bitmap data directly as follows:

def take_screenshot1(hwnd):
wDC = win32gui.GetWindowDC(hwnd)
dcObj=win32ui.CreateDCFromHandle(wDC)
cDC=dcObj.CreateCompatibleDC()
dataBitMap = win32ui.CreateBitmap()
dataBitMap.CreateCompatibleBitmap(dcObj, 500, 500)
cDC.SelectObject(dataBitMap)
cDC.BitBlt((0, 0), (500, 500), dcObj, (0, 0), win32con.SRCCOPY)

im = dataBitMap.GetBitmapBits(True)  # Tried False also
img = np.array(im)
cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
print(img)

dcObj.DeleteDC()
cDC.DeleteDC()
win32gui.ReleaseDC(hwnd, wDC)
win32gui.DeleteObject(dataBitMap.GetHandle())

But i'm not sure how to convert the returned bitmap to a form that OpenCV understands, as there are no methods to convert bitmap to rgb/bgr in OpenCV

In the first version you are reading the entire area of a window. In the second version you are reading 500x500 pixel area. You can't compare the speed unless the 2 areas are the same. There is a bottle neck at `BitBlt`, you can't really do anything about it. — Barmak Shemirani, Oct 18 '16 at 04:04
@Barmak Shermirana Ahh I should have specified better. The window I'm capturing from is 500x500, so the sizes are the same. So I guess I might just have to save and open the img it seems.. — Kyle Hunter, Oct 18 '16 at 09:06
You don't have to save the image data. Just call [GetObject](https://msdn.microsoft.com/en-us/library/windows/desktop/dd144904.aspx) on your *dataBitMap*, and you get direct access to the pixel data through the returned [BITMAP](https://msdn.microsoft.com/en-us/library/windows/desktop/dd183371.aspx) structure. Obvious optimizations include: Create the bitmap and DC outside the screenshot function. And it's not entirely clear, why you call `CreateBitmap` followed by `CreateCompatibleBitmap`. I don't see, why the former is needed. — IInspectable, Oct 18 '16 at 09:28
@IInspectable If I call: win32gui.GetObject(dataBitMap) I get the error "TypeError: The object is not a PyHANDLE object" — Kyle Hunter, Oct 18 '16 at 09:39
That's the wrong `GetObject` call. I was referring to the GDI function (as documented). As an alternative, if you want to store a copy of the bitmap data, you can call [GetDIBits](https://msdn.microsoft.com/en-us/library/windows/desktop/dd144879.aspx) instead. All of the operations you use require somewhat solid understanding of the [Windows GDI](https://msdn.microsoft.com/en-us/library/dd145203.aspx). — IInspectable, Oct 18 '16 at 09:44
@IInspectable I edited post. Using GetBitMapBits I was able to grab the bitmap, but can't convert it after. — Kyle Hunter, Oct 19 '16 at 02:11

score 0 · Answer 1 · edited Jul 09 '21 at 01:56

I'll just show the code that works for me.

import time
import win32gui
import win32ui
import win32con
import win32api
import numpy as np
import cv2


def window_capture():
    hwnd = 0
    hwndDC = win32gui.GetWindowDC(hwnd)
    mfcDC = win32ui.CreateDCFromHandle(hwndDC)
    saveDC = mfcDC.CreateCompatibleDC()
    saveBitMap = win32ui.CreateBitmap()

    MoniterDev = win32api.EnumDisplayMonitors(None, None)
    w = MoniterDev[0][2][2]
    h = MoniterDev[0][2][3]

    saveBitMap.CreateCompatibleBitmap(mfcDC, w, h)
    saveDC.SelectObject(saveBitMap)
    saveDC.BitBlt((0, 0), (w, h), mfcDC, (0, 0), win32con.SRCCOPY)
    im = saveBitMap.GetBitmapBits(True)  # Tried False also
    img = np.frombuffer(im, dtype=np.uint8).reshape((h, w, 4))

    cv2.imshow("demo", img)
    cv2.waitKey(100)


beg = time.time()
for i in range(100):
    window_capture()
end = time.time()
print(end - beg)

cv2.destroyAllWindows()

Could you also explain to us on how this code solves the issue? — Simas Joneliunas, Jul 09 '21 at 02:14

Taking fast screenshot Winapi and Opencv

1 Answers1

Linked