7

I'm trying to apply a blur effect to camera live steam image in ARSCNView. I have checked the WWDC videos. They only mentioned the custom rendering with Metal, but I didn't found any complete example on web. Any idea how to do that?

Updated 1 I have tried to apply a filter to the background. It show incorrect orientation. How can I fix this?

let bg=self.session.currentFrame?.capturedImage

        if(bg != nil){
            let context = CIContext()
            let filter:CIFilter=CIFilter(name:"CIColorInvert")!
            let image:CIImage=CIImage(cvPixelBuffer: bg!)
            filter.setValue(image, forKey: kCIInputImageKey)
            let result=filter.outputImage!
            self.sceneView.scene.background.contents = context.createCGImage(result, from: result.extent)

        }
Wayne Tam
  • 71
  • 1
  • 3
  • 1
    It might be that you can use Post Processing in SceneKit with ScnTechnique; I have not tried it myself. https://developer.apple.com/documentation/scenekit/scntechnique – M. Bedi Jun 19 '17 at 18:59
  • Looks like the the live stream image is set as the scene background and I'm not sure how to add filters to it. You also have access to the pixel buffer through sceneView.session.currentFrame?.capturedImage so you might be able to apply the filter to it and update the background – Guig Jun 19 '17 at 20:33
  • @Guig Thanks! Its work! I can apply some CIFilters on the background now! But the orientation is incorrect... – Wayne Tam Jun 20 '17 at 05:50
  • If you don't mind fiddling with YCbCr pixelbuffers, [this solution works](https://stackoverflow.com/questions/45919745/reliable-access-and-modify-captured-camera-frames-under-scenekit) – diviaki Aug 30 '17 at 12:23

2 Answers2

5

I've found a pretty good solution, which is to simply apply a corresponding geometric transform to the background property whenever the device orientation changes:

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    let image = CIImage(cvPixelBuffer: frame.capturedImage)
    filter.setValue(image, forKey: kCIInputImageKey)

    let context = CIContext()
    if let result = filter.outputImage,
        let cgImage = context.createCGImage(result, from: result.extent) {

        sceneView.scene.background.contents = cgImage
        if let transform = currentScreenTransform() {
            sceneView.scene.background.contentsTransform = transform
        }
    }
}

private func currentScreenTransform() -> SCNMatrix4? {
    switch UIDevice.current.orientation {
    case .landscapeLeft:
        return SCNMatrix4Identity
    case .landscapeRight:
        return SCNMatrix4MakeRotation(.pi, 0, 0, 1)
    case .portrait:
        return SCNMatrix4MakeRotation(.pi / 2, 0, 0, 1)
    case .portraitUpsideDown:
        return SCNMatrix4MakeRotation(-.pi / 2, 0, 0, 1)
    default:
        return nil
    }
}

Make sure you call UIDevice.current.beginGeneratingDeviceOrientationNotifications() in your viewDidLoad method first.

Vladimir Mitrovic
  • 1,780
  • 17
  • 15
  • 1
    Hello, I want to integrate color filter using CIFilter in ARKit, I have tried with above code but at run time, output is getting stretched, i have already posted on it, please have a look, https://stackoverflow.com/questions/58501761/live-camera-is-getting-stretched-while-rendering-using-cifilter-swift-4 – Anand Nanavaty Oct 23 '19 at 05:54
0

I had the same problem. You can try this 

let bg = sceneView.session.currentFrame?.capturedImage
    if (bg != nil) {
        let context = CIContext()
        let filter:CIFilter = CIFilter(name: "CIColorInvert")!

        let image:CIImage = CIImage(cvPixelBuffer: bg!, options: nil)

        let cgImage:CGImage = context.createCGImage(image, from: image.extent)!
        let uiImage:UIImage = UIImage.init(cgImage: cgImage)
        let resultImage = CIImage(image: uiImage)?.oriented(forExifOrientation: imageOrientationToDeviceOrientation(value: UIDevice.current.orientation))

        filter.setValue(resultImage, forKey: kCIInputImageKey)

        let result = filter.outputImage!
        self.sceneView.scene.background.contents = context.createCGImage(result, from: result.extent)
    }


func imageOrientationToDeviceOrientation(value: UIDeviceOrientation) -> Int32 {
            switch (value) {
            case .portrait:
                return 6
            case .landscapeLeft:
                return 1
            case .landscapeRight:
                return 3
            default:
                return 6
            }
        }
Anna Sukhorukova
  • 11
  • 3