Using AVCaptureVideoDataOutputSampleBufferDelegate without a preview window

Question

I'm working on a Swift-based macOS app where I need to capture video input, but not display it on the screen...rather than display the video, I want to send the buffered data for processing elsewhere, and eventually display it on an object in a SceneKit scene.

I have a CameraInput class that has a prepareCamera method:

    fileprivate func prepareCamera() {
        self.videoSession = AVCaptureSession()
        self.videoSession.sessionPreset = AVCaptureSession.Preset.photo

        if let devices = AVCaptureDevice.devices() as? [AVCaptureDevice] {
            for device in devices {
                if device.hasMediaType(AVMediaType.video) {
                    cameraDevice = device

                    if cameraDevice != nil  {
                        do {
                            let input = try AVCaptureDeviceInput(device: cameraDevice)


                            if videoSession.canAddInput(input) {
                                videoSession.addInput(input)
                            }


                           } catch {
                            print(error.localizedDescription)
                        }
                    }
                }
            }

            let videoOutput = AVCaptureVideoDataOutput()
            videoOutput.setSampleBufferDelegate(self as AVCaptureVideoDataOutputSampleBufferDelegate, queue: DispatchQueue(label: "sample buffer delegate", attributes: []))
            if videoSession.canAddOutput(videoOutput) {
                videoSession.addOutput(videoOutput)
            }
        }
    }

And a startSession method that starts the AVCaptureSession session:

fileprivate func startSession() {
    if let videoSession = videoSession {
        if !videoSession.isRunning {
            self.videoInputRunning = true
            videoSession.startRunning()
        }
    }
}

I also implement AVCaptureVideoDataOutputSampleBufferDelegate, where I intend to capture the CMSampleBuffer for later use:

extension CameraInput: AVCaptureVideoDataOutputSampleBufferDelegate {

    internal func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
        print(Date())
    }
}

However, the delegate is never called. Is this a situation where I have to display the video output in order for this to be called?

Under what class are you writing these lines of code? And what is your input device? — El Tomato, Dec 05 '17 at 22:05
@ElTomato the input device is just my mac's iSIght camera. The class is a custom class called CameraInput that inherits from NSObject. — narner, Dec 05 '17 at 22:07
You are not using previewLayer. What is it for? To my knowledge, you do need it. — El Tomato, Dec 05 '17 at 22:12
@ElTomato Agh, yes; that was left in there by accident - I updated my code above. — narner, Dec 05 '17 at 22:13

rickster · Accepted Answer · 2017-12-05T23:19:33.677

None of your issues have to do with whether (or not) you display a preview of captured video.

If you're on Swift 4 (and it looks like you are), the delegate method signature you're looking to implement isn't captureOutput(_:didOutputSampleBuffer:from:), it's this:

optional func captureOutput(_ output: AVCaptureOutput, 
              didOutput sampleBuffer: CMSampleBuffer, 
                     from connection: AVCaptureConnection)

Unrelated tips:

Namespaced constants mean you can be more brief if you like; e.g. videoSession.sessionPreset = .photo
AVCaptureDevice.devices() is deprecated. Instead of calling that and looping through devices yourself, just ask AVCaptureDevice for exactly the kind of device you want:
```
let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, 
                                       for: .video, position: .back)
```
You don't need the as cast in videoOutput.setSampleBufferDelegate(self as AVCaptureVideoDataOutputSampleBufferDelegate if your class already declares conformance to the AVCaptureVideoDataOutputSampleBufferDelegate protocol.

Finally, if you're just looking to get the live-from-camera video mapped onto some part of a SceneKit scene, note that in iOS 11 you can assign an AVCaptureDevice to an SCNMaterialProperty's contents directly — no need to grab, process, and move pixel buffers yourself.

Wow, I really wish SpriteKit had that same `contents` feature. I'm trying to show a webcam in a SKNode, and it's looking like I have to copy each frame into a texture. — bugloaf, Sep 08 '22 at 19:52

Using AVCaptureVideoDataOutputSampleBufferDelegate without a preview window

1 Answers1

Linked