Convert HBITMAP to cv::Mat

Question

I can't seem to figure out how to do this, I have searched on google and found two code examples, one from a github screen capture library and another on a posting group and neither of them seem to be working.

I have a struct:

struct ClacksScreen
{
    HWND hDesktopWnd; 
    int width, height;
    RECT wr, cr;
    HDC hdcClacksScreen; // hardware ClacksScreen
    HDC hdcMemDC; // ClacksScreen in memory
    HBITMAP hbmClacksScreen; //hbitmap of the ClacksScreen
    BITMAP bmpClacksScreen;
    BITMAPINFOHEADER bi;
};

This is updated. I have some functions defined, including one that writes the bitmap to disk, this works fine, the screen is captured and a bmp is written to disk and it's the one I wanted.

Now I want to convert the HBITMAP I take of the screen directly into a cv::Mat for OpenCV2.1.

It kind of works, except the image is pure grey and it crashes. Obviously I am still pretty n00b when it comes to c++ so there is probably something simple that I am just not groking.

static cv::Mat copyToCVMat(const ClacksScreen * s)
{
    cv::Mat image;
    image.create(s->bmpClacksScreen.bmWidth, s->bmpClacksScreen.bmHeight, CV_8UC4);
    GetDIBits(s->hdcMemDC, s->hbmClacksScreen, 0,
            (UINT)s->bmpClacksScreen.bmHeight,
            image.data,
            (BITMAPINFO *)&s->bi, DIB_RGB_COLORS);
    return image;
}

When I wrap a cv::imwrite(image); in a try catch, I get a bad allocation error. Obviously at this point we've established that I have no frelling clue how to do this, so any help would be appreciated.

UPDATE

If I run this code:

try {
    cv::Mat screen = cv::imread("captureqwsx.jpg");
    if (!screen.data) {
        printf("no image data?");
    }
    cv::imwrite("out.jpg",screen);
} catch(std::exception e) {
    printf("Exception %s\n",e.what());
}

I get the output:

no image data? Exception bad allocation

When I try to run the high gui, it's the same as before, problem crops up for both .jpg and .bmps written to disk, which are viewable in image viewer and MS Paint fine.

I tried with a completely different image, a .png from a website, same issue.

So what am I doing wrong at this point?

Answer 1

I wrote my own function to do something similar, hopefully it can help you: How to capture the desktop in OpenCV (ie. turn a bitmap into a Mat)?

Answer 2

From the OpenCV documentation

data – Pointer to the user data. Matrix constructors that take data and step parameters do not allocate matrix data. Instead, they just initialize the matrix header that points to the specified data, i.e. no data is copied. This operation is very efficient and can be used to process external data using OpenCV functions. The external data is not automatically deallocated, user should take care of it.

I think this is the point, using this constructor does not copy the buffer you are passing to the constructor, so you should not free this data until you no longer need the cv::Mat. Also I know next to nothing about this stuff, but why are you passing the BITMAPINFOHEADER and BITMAPFILEHEADER to your cv::Mat object, that doesn't seem right at all.

Answer 3

I don't know what it is, but the minute I ask a question, I get obssesed with finding the answer. Anyway, part of this issue is actually solved by the answer to this question:

OpenCV 2.0 C++ API using imshow: returns unhandled exception and "bad-flag"

In Visual C++:

go to Project->Properties (or Alt-F7) Configuration Properties->Linker->Input->Additional Dependencies

replace the usual " cv210.lib cxcore210.lib highgui210.lib" by " cv210d.lib cxcore210d.lib highgui210d.lib" - which are the debugging libraries.

highgui still shows grey and doesn't work, but reading from the HBITMAP using the above method now works. and I don't really need highgui, it was just for testing anyway.

Answer 4

This code is even more reliable than the same OpenCV library. In OpenCV there is a specific function to load HBITMAPS, but it has some problems, such as loading the image backwards, I prefer to have control of my code.

cv::Mat LoadHBITMAPOpenCV(HBITMAP& HBitmap, Gdiplus::Rect rect = Gdiplus::Rect())
{
    // Create a Bitmap object from the HBITMAP, avoid memory leaks
    std::unique_ptr<Bitmap> bitmap(Bitmap::FromHBITMAP(HBitmap, NULL));
    if (!bitmap)
    {
        // Error loading the HBITMAP
        throw std::runtime_error("[LoadHBITMAPOpenCV] Error loading HBITMAP");
    }

    // Get the bitmap dimensions
    int bitmapWidth = bitmap->GetWidth();
    int bitmapHeight = bitmap->GetHeight();

    // Check if the bitmap is valid
    if (bitmapWidth <= 0 || bitmapHeight <= 0)
    {
        throw std::runtime_error("[LoadHBITMAPOpenCV] Corrupt HBITMAP");
    }

    int rectWidth = bitmapWidth;
    int rectHeight = bitmapHeight;
    // Check if a rect is provided, otherwise use the entire image as the rect
    if (rect.IsEmptyArea())
    {
        rect = Gdiplus::Rect(0, 0, bitmapWidth, bitmapHeight);
    }
    else
    {
        // Check if the cropping rectangle is valid
        rectWidth = rect.Width - rect.X;
        rectHeight = rect.Height - rect.Y;
        if (rectWidth <= 0 || rectHeight <= 0 || rect.X < 0 || rect.Y < 0 || rect.Width > bitmapWidth || rect.Height > bitmapHeight)
        {
            throw std::runtime_error("[LoadHBITMAPOpenCV] Invalid rect");
        }
    }

    // Load the HBITMAP into cv::Mat using GDI+
    cv::Mat mat(rectHeight, rectWidth, CV_8UC4);

    BitmapData bitmapData;
    Rect rect_cut(rect.X, rect.Y, rectWidth, rectHeight);
    bitmap->LockBits(&rect_cut, ImageLockModeRead, PixelFormat32bppARGB, &bitmapData);

    if (bitmapData.Stride < 0)
    {
        bitmap->UnlockBits(&bitmapData);
        throw std::runtime_error("[LoadHBITMAPOpenCV] Corrupt HBITMAP stride");
    }

    BYTE* data = reinterpret_cast<BYTE*>(bitmapData.Scan0);
    memcpy(mat.data, data, mat.total() * mat.elemSize());

    bitmap->UnlockBits(&bitmapData);

    cv::cvtColor(mat, mat, EXTERNAL_CTV_COLOR);
    // cv::cvtColor(mat, mat, cv::COLOR_RGB2Lab);

    return mat;
}