How to convert CMSampleBuffer to OpenCV's Mat instance in swift

Question

I am working on an OpenCV project where I am taking input from iPhone native camera as CMSampleBuffer now I wanted to create Mat instance that is required in OpenCV for further process

I have found some old post related with it but all are not working in current swift as those are pretty old.

Raw image data from camera like "645 PRO"

How to convert CMSampleBufferRef to IplImage (iOS)

sagarthecoder · Accepted Answer · 2022-04-28T06:01:32.427

1

First, convert CMSampleBuffer To UIImage.

extension CMSampleBuffer {
    func asUIImage()-> UIImage? {
        guard let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(self) else {
            return nil
        }
        let ciImage = CIImage(cvPixelBuffer: imageBuffer)
        return convertToUiImage(ciImage: ciImage)
    }
    
    func convertToUiImage(ciImage: CIImage) -> UIImage? {
        let context = CIContext(options: nil)
        context.clearCaches()
        guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
            return nil
        }
        let image = UIImage(cgImage: cgImage)
        return image
    }
}

Then you can easily convert UIImage to Mat and return UIImage with/without doing something.
OpenCVWrapper.h file

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

NS_ASSUME_NONNULL_BEGIN

@interface OpenCVWrapper : NSObject

+ (UIImage *) someOperation : (UIImage *) uiImage;

@end

NS_ASSUME_NONNULL_END

OpenCVWrapper.mm file

#import <opencv2/opencv.hpp>
#import <opencv2/imgcodecs/ios.h>
#import <opencv2/core/types_c.h>
#import "OpenCVWrapper.h"
#import <opencv2/Mat.h>

#include <iostream>
#include <random>
#include <vector>
#include <chrono>
#include <stdint.h>

@implementation OpenCVWrapper

+ (UIImage *) someOperation : (UIImage *) uiImage {
    cv::Mat sourceImage;
    UIImageToMat(uiImage, sourceImage);
    // Do whatever you need
    return MatToUIImage(sourceImage);
}

@end

edited Apr 28 '22 at 06:01

answered Apr 17 '22 at 05:56

sagarthecoder

137
1
9

Thanks for the answer but can you please specify where to put `getMatInstance`'method `.cpp` file or the `.mm` file? Also what I need to put in the header file, I am pretty new to objective C and C++ – Sajal Gupta Apr 21 '22 at 02:26
You need to use ```getMatInstance``` in ```.mm``` file. Read this https://medium.com/@yiweini/opencv-with-swift-step-by-step-c3cc1d1ee5f1 – sagarthecoder Apr 21 '22 at 05:14
I updated my code. please check it again. – sagarthecoder Apr 21 '22 at 05:27
As per your answer I have updated my code but after doing the same I am getting two errors 1- "OpenCV 4.x+ requires enabled C++11 support" 2- 'array' file not found there is another relatable question I have found at stackoverflow but it seems like they were using some old version's API however I am using the latest one then how come the errors are coming https://stackoverflow.com/questions/61008411/xcode-opencv-4-x-requires-enabled-c11-support do you have any idea ? – Sajal Gupta Apr 25 '22 at 08:50
I didn't know about that. Maybe you wrongly integrated openCV into your project. I followed this(https://medium.com/@yiweini/opencv-with-swift-step-by-step-c3cc1d1ee5f1) and successfully integrated it without any errors. – sagarthecoder Apr 25 '22 at 18:11
okay no problem, BTW which version you have used in your project, in the above article Yiwei Ni have used 3.2.0 and I am using latest version 4.5.5 – Sajal Gupta Apr 26 '22 at 05:54
I used the same version(4.5.5). Integrated it 2 weeks ago. – sagarthecoder Apr 26 '22 at 07:37
I have created a new project and integrated the openCV framework as explained in this article [link] (medium.com/@yiweini/opencv-with-swift-step-by-step-c3cc1d1ee5f1) and I am able to print version in console but as I added below line in `.h` file then I am getting then I am getting `expected a type error` at following line `+ (cv::Mat) getMatInstance : (UIImage *) uiImage;` Here is the screenshot of the error [link](https://drive.google.com/file/d/1xYnfIvAHMFlNN6cBTCQrWGt1zebpoz8L/view?usp=sharing) – Sajal Gupta Apr 28 '22 at 05:39
Do you need ```Mat``` instance outside of OpenCVWrapper.mm class? – sagarthecoder Apr 28 '22 at 05:51
I updated my code again. check it. – sagarthecoder Apr 28 '22 at 06:01
Thanks for your answer it's partially solved my problem but I wanted to use Mat from my swift class, is it possible? in simple words `I need Mat instance outside of OpenCVWrapper.mm` – Sajal Gupta May 04 '22 at 08:42
As far as I know ```Mat``` is used as a data type of C++. It's not allowed outside of ```.mm``` file hence ```.mm``` file refered to C++ file. I don't know for which purpose you need it in swift class. – sagarthecoder May 05 '22 at 13:45
Can you please answer this question [StackOverflow](https://stackoverflow.com/questions/72168170/how-to-convert-cv-mat-to-objective-c-mat-in-opencv) – Sajal Gupta May 09 '22 at 07:14

score 0 · Answer 2 · answered Apr 14 '22 at 05:55

As far as I know openCV is written in C++ and you will have to interact with it using an Objc++ or an Objc wrapper. Hence the resources you found are solid and there are many others on working with OpenCV on iOS. A simple swifty wrapper I found online is - https://github.com/Legoless/LegoCV.

There is a full interoperability between Objc/Objc++ and swift and it is fairly easy to implement. See these two resources for more info - https://developer.apple.com/documentation/swift/imported_c_and_objective-c_apis/importing_objective-c_into_swift https://rderik.com/blog/understanding-objective-c-and-swift-interoperability/

How to convert CMSampleBuffer to OpenCV's Mat instance in swift

2 Answers2