Reading a PyCBitmap with OpenCV

Question

I created an image from a window screenshot using Win32gui. The object has the type:

object 'PyCBitmap' - assoc is 000002AF9A64DB50, vi=<None>

I want to then pass this for analysis with OpenCV. I have had success reading in a saved .bmp file using:

cv2.imread(img_file, 0)

When trying using cv2.imread for a PyCBitmap object I get the following error:

TypeError: bad argument type for built-in operation

My question is:

How can I convert the PyCBitmap object into an acceptable type for cv2.imread, without having to save the object as a .bmp file first?

Thanks in advance,

Behzad

p.s I'm using opencv 3.1 with python bindings, I'm happy to follow advice written in C++ or python :)

Answer 1

I've been looking for the same thing, and I finally found it by combining several other SO answers:

• PIL and Bitmap from WinAPI
• https://stackoverflow.com/a/14140796/343381

Basically, the code I came up with is:

import PIL, numpy, cv2
bmpinfo = dataBitMap.GetInfo()
bmparray = numpy.asarray(dataBitMap.GetBitmapBits(), dtype=numpy.uint8)
pil_im = Image.frombuffer('RGB', (bmpinfo['bmWidth'], bmpinfo['bmHeight']), bmparray, 'raw', 'BGRX', 0, 1)
pil_array = numpy.array(pil_im)
cv_im = cv2.cvtColor(pil_array, cv2.COLOR_RGB2BGR)

Brief explanation: Python OpenCV just uses numpy arrays, so the trick is really getting the bytes into the right numpy array format. As it turns out, for this you need an image processing library like PIL that can handle the image-specific logic like cutting out the alpha channel. The input data is generally RGBX format, PIL converts to RGB, and OpenCV converts that to BGR which it likes.

I profiled this and unfortunately it is dramatically slower than GetBitmapBits() and in converting its tuple result to an array.

Answer 2

I know i'm VERY late but maybe itll help someone in the future as this took WAY to long to find. This is a MASSIVE increase in speed. For some reason numpy.asarray() and alot of the other numpy array creation methods are VERY slow. All my findings are mostly based off Adam S answer.

the line

bmparray = numpy.array(bitmap.GetBitmapBits()).astype(numpy.uint8)

on my machine take about 0.25 seconds which is not very nice. To fix this we can avoid creating an array here.

bmpinfo = bitmap.GetInfo()
bmpbits = bitmap.GetBitmapBits(True)
pil_im = Image.frombuffer('RGB', (bmpinfo['bmWidth'], bmpinfo['bmHeight']), bmpbits, 'raw', 'BGRX', 0, 1)
pil_array = numpy.array(pil_im)    
img = cv2.cvtColor(pil_array, cv2.COLOR_RGB2BGR)

here is my optimized version of the code, it can take ~55 screenshots in 1 seconds, as appose to the old methods getting ~6 per second:

import win32gui
import win32ui
import win32con
import numpy, cv2
from PIL import Image

def convert(bitmap) -> cv2.Mat:
    bmpinfo = bitmap.GetInfo()
    bmpbits = bitmap.GetBitmapBits(True)
    pil_im = Image.frombuffer('RGB', (bmpinfo['bmWidth'], bmpinfo['bmHeight']), bmpbits, 'raw', 'BGRX', 0, 1)
    pil_array = numpy.array(pil_im)    
    return cv2.cvtColor(pil_array, cv2.COLOR_RGB2BGR)


def take_screen_shot():
    # https://stackoverflow.com/questions/3586046/fastest-way-to-take-a-screenshot-with-python-on-windows
    w = 1920 # set this
    h = 1080 # set this
    
    hwnd = win32gui.FindWindow(None, "filler")
    wDC = win32gui.GetWindowDC(hwnd)
    dcObj=win32ui.CreateDCFromHandle(wDC)
    cDC=dcObj.CreateCompatibleDC()
    dataBitMap = win32ui.CreateBitmap()
    dataBitMap.CreateCompatibleBitmap(dcObj, w, h)
    cDC.SelectObject(dataBitMap)
    cDC.BitBlt((0,0),(w, h) , dcObj, (0,0), win32con.SRCCOPY)
    
    image = convert(dataBitMap)

    # Free Resources
    dcObj.DeleteDC()
    cDC.DeleteDC()
    win32gui.ReleaseDC(hwnd, wDC)
    win32gui.DeleteObject(dataBitMap.GetHandle())
    return image

---

Edit:

Okay I got curious and ended up using GPT3 to assist to see if could come up with the same answer (with some assistance) and holy smokes did it make a difference.

using this code:

def take_screenshot():
    # Find the handle for the window named "filler"
    hwnd = win32gui.FindWindow(None, "filler")

    # Get the dimensions of the window
    left, top, right, bottom = win32gui.GetClientRect(hwnd)
    w = right - left # set size here if you want
    h = bottom - top # set size here if you want 

    # Create a device context (DC) for the window
    hwndDC = win32gui.GetWindowDC(hwnd)
    mfcDC = win32ui.CreateDCFromHandle(hwndDC)
    dc = mfcDC.CreateCompatibleDC()

    # Create a bitmap object and select it into the device context
    bmp = win32ui.CreateBitmap()
    bmp.CreateCompatibleBitmap(mfcDC, w, h)
    dc.SelectObject(bmp)

    # Copy the contents of the window into the bitmap object
    dc.BitBlt((0, 0), (w, h), mfcDC, (0, 0), win32con.SRCCOPY)

    # Convert the bitmap object to a numpy array
    signedIntsArray = bmp.GetBitmapBits(True)
    img = np.frombuffer(signedIntsArray, dtype='uint8')
    img.shape = (h, w, 4)

    # Clean up
    dc.DeleteDC()
    mfcDC.DeleteDC()
    win32gui.ReleaseDC(hwnd, hwndDC)
    win32gui.DeleteObject(bmp.GetHandle())

    return img

this code allowed me personally to take screenshots at ~100 per second