Convert VNRectangleObservation points to other coordinate system

Question

I need to convert the VNRectangleObservation received CGPoints (bottomLeft, bottomRight, topLeft, topRight) to another coordinate system (e.g. a view's coordinate on screen).

I define a request:

    // Rectangle Request
    let rectangleDetectionRequest = VNDetectRectanglesRequest(completionHandler: handleRectangles)
    rectangleDetectionRequest.minimumSize = 0.5
    rectangleDetectionRequest.maximumObservations = 1

I get the sampleBuffer from camera in delegate call, and perform a detection request:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {return}
    var requestOptions:[VNImageOption:Any] = [:]
    if let cameraIntrinsicData = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil) {
        requestOptions = [.cameraIntrinsics:cameraIntrinsicData]
    }
    let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: CGImagePropertyOrientation(rawValue:6)!, options: requestOptions)
    do {
        try imageRequestHandler.perform(self.requests)
    } catch {
        print(error)
    }

}

Later in completionHandler I receive the results:

func handleRectangles (request:VNRequest, error:Error?) {

     guard let results = request.results as? [VNRectangleObservation] else { return }

     let flipTransform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -self.previewView.frame.height)
     let scaleTransform = CGAffineTransform.identity.scaledBy(x: self.previewView.frame.width, y: self.previewView.frame.height)

     for rectangle in results {
        let rectangleBounds = rectangle.boundingBox.applying(scaleTransform).applying(flipTransform)
        // convertedTopLeft = conversion(rectangle.topLeft) 
        // convertedTopRight = conversion(rectangle.topRight) 
        // convertedBottomLeft = conversion(rectangle.bottomLeft) 
        // convertedBottomRight = conversion(rectangle.bottomRight) 
    }
}

This works for boundingBox which is CGRect, but I need to transform the CGPoints instead to a coordinate system of another view. The problem is that I don't know how to get the transformation from the sampleBuffer: CMSampleBuffer's coordinate system to the previewView coordinate system.

Thanks!

score 8 · Accepted Answer · answered Jan 18 '18 at 20:54

That was simply a matter of applying the transform to the CGPoint itself where size is the CGSize of the destination view for which I need transpose the four points.

    let transform = CGAffineTransform.identity
        .scaledBy(x: 1, y: -1)
        .translatedBy(x: 0, y: -size.height)
        .scaledBy(x: size.width, y: size.height)

    let convertedTopLeft = rectangle.topLeft.applying(transform)
    let convertedTopRight = rectangle.topRight.applying(transform)
    let convertedBottomLeft = rectangle.bottomLeft.applying(transform)
    let convertedBottomRight = rectangle.bottomRight.applying(transform)

that's perfect but how do I remove this transform again by little modifying its corners? I need to pass those points again to CIPerspectiveCorrection. — Bhushan B, Oct 16 '18 at 10:58

score 2 · Answer 2 · answered Mar 06 '19 at 16:46

@mihaicris answer works, but only in portrait mode. In landscape, we need to do it a little different.

if UIApplication.shared.statusBarOrientation.isLandscape {
    transform = CGAffineTransform.identity
        .scaledBy(x: -1, y: 1)
        .translatedBy(x: -size.width, y: 0)
        .scaledBy(x: size.width, y: size.height)
} else {
    transform = CGAffineTransform.identity
        .scaledBy(x: 1, y: -1)
        .translatedBy(x: 0, y: -size.height)
        .scaledBy(x: size.width, y: size.height)
}

let convertedTopLeft = rectangle.topLeft.applying(transform)
let convertedTopRight = rectangle.topRight.applying(transform)
let convertedBottomLeft = rectangle.bottomLeft.applying(transform)
let convertedBottomRight = rectangle.bottomRight.applying(transform)

score 1 · Answer 3 · answered Jan 09 '18 at 09:12

1

I assume you use layer for the camera, and the layer is AVCaptureVideoPreviewLayer. (https://developer.apple.com/documentation/avfoundation/avcapturevideopreviewlayer). So if you want to convert single point, use this function:layerPointConverted (https://developer.apple.com/documentation/avfoundation/avcapturevideopreviewlayer/1623502-layerpointconverted). Please notices that the y is inverted because of the VNRectangleObservation coordinates system.

let convertedTopLeft: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.topLeft.x, y: 1 - rectangle.topLeft.y))
let convertedTopRight: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.topRight.x, y: 1 - rectangle.topRight.y))
let convertedBottomLeft: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.bottomLeft.x, y: 1 - rectangle.bottomLeft.y))
let convertedBottomRight: CGPoint = cameraLayer.layerPointConverted(fromCaptureDevicePoint: CGPoint(x: rectangle.bottomRight.x, y: 1 - rectangle.bottomRight.y))

Hope it helped

answered Jan 09 '18 at 09:12

Tziki

311
1
8

Thanks I'll check in my code your answer because it might be a better solution to compensate the aspect fill gravitypoint property of the preview layer. – mihaicris Jan 18 '18 at 21:00
Cool, let me know if it helped – Tziki Jan 21 '18 at 10:14
1

Hi, I tried the layerPointConverted method, but it doesn't work as expected. With input points as corners of capture device coordinates (0,0 - 1,0 - 0,1 - 1,1) the converted points x coordinates are off the destination view frame. Is like the method doesn't take into account the aspectFill videogravity property of the previewView (which is exactly as the screen size, not bigger). I thought this function knows how to compensate the offset.. – mihaicris Jan 31 '18 at 09:42
@mihaicris I'm having the same issue, x coordinates are off the destination view frame. But, if you go to Apple documentation (https://developer.apple.com/documentation/avfoundation/avcapturevideopreviewlayer/1623502-layerpointconverted), it says that "The conversion performed by this method takes the layer’s frame size and the receiver’s videoGravity property into consideration." I will investigate this issue and see if I'm able to figure out what's happening – kikettas Aug 16 '18 at 09:34

Convert VNRectangleObservation points to other coordinate system

3 Answers3