Cannot figure out Numpy equivalent for cv.mat.step[0]

Question

I am currently in the process of transferring code from an old OpenCV example into OpenCV3 in Python (using PyObjC and the Quartz module). The Objective-C code takes a UIImage and creates a material that can be used by OpenCV. My python code takes a CGImage and does the same thing.

Here is the Objective-C code:

(cv::Mat)cvMatFromUIImage:(UIImage *)image {
  CGColorSpaceRef colorSpace = CGImageGetColorSpace(image.CGImage);
  CGFloat cols = image.size.width;
  CGFloat rows = image.size.height;

  cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels (color channels + alpha)

  CGContextRef contextRef = CGBitmapContextCreate(cvMat.data,                 // Pointer to  data
                                                 cols,                       // Width of bitmap
                                                 rows,                       // Height of bitmap
                                                 8,                          // Bits per component
                                                 cvMat.step[0],              // Bytes per row
                                                 colorSpace,                 // Colorspace
                                                 kCGImageAlphaNoneSkipLast |
                                                 kCGBitmapByteOrderDefault); // Bitmap info flags

  CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), image.CGImage);
  CGContextRelease(contextRef);

  return cvMat;
}

Here is my Python equivalent:

def macToOpenCV(image):
    color_space = CGImageGetColorSpace(image)
    column = CGImageGetHeight(image)
    row = CGImageGetWidth(image)
    mat = np.ndarray(shape=(row, column, 4), dtype=np.uint8)
    c_ref = CGBitmapContextCreate(mat,
                                  row,
                                  column,
                                  8,
                                  ,  # mat.step[0],
                                  color_space,
                                  kCGImageAlphaNoneSkipLast |
                                  kCGBitmapByteOrderDefault)

    CGContextDrawImage(c_ref, CGRectMake(0, 0, column, row), image)
    return mat

I am fairly confident that I have most of this right currently, but I am lost what I should be calling for the equivalent of cvMat.step[0] in Numpy. I also would welcome some general code review on the code segment, because when I use cv2.imshow() I am not getting the image I expect at all :).

Thanks!

score 1 · Accepted Answer · edited May 23 '17 at 11:52

I ended up abandoning the above approach and found an answer on this stack overflow question that worked after a little bit of editing: Converting CGImage to python image (pil/opencv)

image_ref = CGWindowListCreateImage(CGRectNull,
                                        kCGWindowListOptionIncludingWindow,
                                        wid,
                                        kCGWindowImageBoundsIgnoreFraming)

pixeldata = CGDataProviderCopyData(CGImageGetDataProvider(image_ref))

height = CGImageGetHeight(image_ref)
width = CGImageGetWidth(image_ref)

image = Image.frombuffer("RGBA", (width, height),
                         pixeldata, "raw", "RGBA", 0, 1)

# Color correction from BGRA to RGBA
b, g, r, a = image.split()
image = Image.merge("RGBA", (r, g, b, a))
np.array(image)
return np.array(image)

Image in this case is PIL.Image. You also can see I opted for an automatic stride calculation (parameter 0 in frombuffer()) mostly because the function the answer used I could not get to work.

score 0 · Answer 2 · answered Feb 06 '20 at 08:44

From buffer is fine, but pictures of widths which are not multiples of 64 will be screwed up. See this one. I made an efficient approach using nps frombuffer and as_strides which skips the dead pixels:

cg_img = CG.CGWindowListCreateImage(
    CG.CGRectNull,
    CG.kCGWindowListOptionIncludingWindow,
    wnd_id,
    CG.kCGWindowImageBoundsIgnoreFraming | CG.kCGWindowImageNominalResolution
)

bpr = CG.CGImageGetBytesPerRow(cg_img)
width = CG.CGImageGetWidth(cg_img)
height = CG.CGImageGetHeight(cg_img)

cg_dataprovider = CG.CGImageGetDataProvider(cg_img)
cg_data = CG.CGDataProviderCopyData(cg_dataprovider)

np_raw_data = np.frombuffer(cg_data, dtype=np.uint8)

np_data = np.lib.stride_tricks.as_strided(np_raw_data,
                                          shape=(height, width, 3),
                                          strides=(bpr, 4, 1),
                                          writeable=False)

Cannot figure out Numpy equivalent for cv.mat.step[0]

2 Answers2