Converting UIImage to cv::Mat

Question

I have a UIImage which is a pic captured from an iPhone Camera now I want the UIImage to be converted to cv::Mat (OpenCV). I am using the following lines of code to accomplish this :

-(cv::Mat)CVMat
{

CGColorSpaceRef colorSpace = CGImageGetColorSpace(self.CGImage);
CGFloat cols = self.size.width;
CGFloat rows = self.size.height;

cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels

CGContextRef contextRef = CGBitmapContextCreate(cvMat.data,                 // Pointer to backing data
                                                cols,                      // Width of bitmap
                                                rows,                     // Height of bitmap
                                                8,                          // Bits per component
                                                cvMat.step[0],              // Bytes per row
                                                colorSpace,                 // Colorspace
                                                kCGImageAlphaNoneSkipLast |
                                                kCGBitmapByteOrderDefault); // Bitmap info flags

CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), self.CGImage);
CGContextRelease(contextRef);

return cvMat;
}

This Code works fine for the UIImage in Landscape mode but when I use the same code with Images taken from Portrait mode the images gets rotated by 90 degrees towards the right.

I am a newbie in iOS and Objective C and hence I am not able to figure out what is wrong.

Can somebody tell me what is wrong with the code or am I missing out something.

score 45 · Accepted Answer · edited May 02 '16 at 20:54

45

This will be because the UIImage is not actually portrait. All photos taken with the iPhone camera are landscape in their raw bitmap state, eg 3264 wide x 2488 high. A "portrait" photo is displayed as such by the orientation EXIF flag set in the image, which is honoured, for example, by the photo library app which swivels images according to this flag and the viewing orientation of the camera.

The flag also affects how UIImage reports its width and height properties, transposing them from their bitmap values for images flagged as portrait.

cv::Mat doesn't bother with any of that. This means that (i) when translating to cv::Mat a portrait image will have its size.width and size.height values transposed, and (ii) when translating back from cv::Mat you will have lost the orientation flag.

The simplest way to handle this when going from UIImage to cv::Mat is to swap width and height values if the image is flagged as portrait:

if  (self.imageOrientation == UIImageOrientationLeft
      || self.imageOrientation == UIImageOrientationRight) {
         cols = self.size.height;
         rows = self.size.width;
    }

When translating back from cv::Mat to UIImage, you will want to reinstate the orientation flag. Assuming your cv::Mat -> UIImage code contains this:

self = [self initWithCGImage:imageRef];

you can use this method instead, and reset the orientation as per the original.

self = [self initWithCGImage:imageRef scale:1 orientation:orientation];

edited May 02 '16 at 20:54

shim

9,289
12
69
108

answered Jan 15 '13 at 10:52

foundry

31,615
9
90
125

can you provide a complete sample ? what's the cols and rows above ? – tomriddle_1234 Jul 06 '15 at 11:27
@tomriddle_1234 - the code in my answer is designed to be added to the code in the question, before `CGBitmapContextCreate` is called. That function uses `cols` and `rows` to specify the dimensions of the bitmap context. – foundry Jul 06 '15 at 12:16
thanks, I missed the question code. I found another solution of his question, he can normalize the image as http://stackoverflow.com/a/10611036/921082 – tomriddle_1234 Jul 11 '15 at 04:56
@tomriddle_1234 - that solution is fine if you are just trying to rotate the image - but if you are also converting to a different bitmap format (as per this question) you will end up drawing a bitmap twice, which is needlessly inefficient. – foundry Jul 11 '15 at 10:08
You are right, I knew there is the UIImageToMat function in OpenCV, and it doesn't handle such EXIF info, this problem confused me since I'm from the OpenCV background rather than iOS. Normalization probably is easier to understand than the alternations of the UIImageToMat. – tomriddle_1234 Jul 11 '15 at 11:17
@foundry I have asked a question somewhat similar to this one regarding an error when building OpenCV for iOS. I would really appreciate it if you could take a look at the question: https://stackoverflow.com/questions/45534183/opencv-error-assertions-failed-on-ios – fi12 Aug 07 '17 at 12:28
`CGFloat cols = CGImageGetWidth(image.CGImage), rows = CGImageGetHeight(image.CGImage);` Opencv builtin uses these for image height and width, it's more elegant, I think. – Itachi Nov 12 '19 at 08:03

score 17 · Answer 2 · answered Mar 08 '17 at 10:05

You should consider using native OpenCV functions to convert forth and back :

#import <opencv2/imgcodecs/ios.h>
...
UIImage* MatToUIImage(const cv::Mat& image);
void UIImageToMat(const UIImage* image,
                         cv::Mat& m, bool alphaExist = false);

Note: if your UIImage comes from the camera, you should 'normalize' it ( iOS UIImagePickerController result image orientation after upload) before converting to cv::Mat since OpenCV does not take into account Exif data. If you don't do that the result should be misoriented.

score 2 · Answer 3 · answered Nov 08 '21 at 14:06

I was trying to convert a portrait UIImage to Mat and back, but I had problems with both approaches of this post.

The native approach with MatToUIImage and UIImageToMat resulted in a rotated image.
The custom approach with CGContext ended up in a strange color bug (not only RGB / BGR swap). Red was red, blue was pink, green was yellow.

I ended up using the native OpenCV with a rotation:

#import <opencv2/imgcodecs/ios.h>

- (cv::Mat)convertUIImageToCVMat:(UIImage *)image
{
    cv::Mat imageMat;
    UIImageToMat(image, imageMat);
    cv::rotate(imageMat, imageMat, cv::ROTATE_90_CLOCKWISE);
    return imageMat;
}

- (UIImage *)convertCVMatToUIImage:(cv::Mat)cvMat
{
    UIImage *image = MatToUIImage(cvMat);
    return image;
}

Converting UIImage to cv::Mat

3 Answers3

Linked