0

I am trying to capture my system's screen and process the data. But I get nil value for CMSampleBufferGetDataBuffer for the sample buffer I get in captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) delegate method. Any idea? below is my code:

import Cocoa
import AVFoundation

class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate {

    private lazy var sampleBufferDelegateQueue = DispatchQueue(label: "VideoCapture")
    private lazy var captureSession: AVCaptureSession = {
        let session = AVCaptureSession()
        session.sessionPreset = .hd1280x720

        if let input = AVCaptureScreenInput.init(displayID: CGMainDisplayID()) {
            session.addInput(input)
        }


        let output = AVCaptureVideoDataOutput()
        output.videoSettings = [
            kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
            kCVPixelBufferMetalCompatibilityKey as String: true
        ]
        output.setSampleBufferDelegate(self, queue: self.sampleBufferDelegateQueue)
        session.addOutput(output)

        return session
    }()


    @IBAction func startAction(_ sender: Any) {
        self.start()
    }
    override func viewDidLoad() {
        super.viewDidLoad()

    }

    func start() {
        guard !self.captureSession.isRunning else {
            return
        }

        self.captureSession.startRunning()
    }

    func stop() {
        guard self.captureSession.isRunning else {
            return
        }

        self.captureSession.stopRunning()
    }


    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {


         let blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
        print(blockBuffer ?? "nil")

    }
}
prabhu
  • 1,158
  • 1
  • 12
  • 27
  • whats in sampleBuffer? – Daniel Apr 07 '20 at 15:58
  • also if you look at CMSampleBufferGetDataBuffer ref -- CMBlockBuffer of media data. The result will be NULL if the CMSampleBuffer does not contain a CMBlockBuffer, if the CMSampleBuffer contains a CVImageBuffer, or if there is some other error, do you ahve an image buffer there? – Daniel Apr 07 '20 at 16:06
  • Sample buffer contains the data for a frame captured with a AVCaptureSession. Yes the will be NULL if CMSamoleBuffer does not contain the bolck buffer and thats what happening here. I want to know why it doesn't contain the block buffer. I need the block buffer for my purpose. – prabhu Apr 07 '20 at 16:10
  • is the frame a CVImageBuffer? if so check https://stackoverflow.com/questions/45038919/how-do-i-convert-a-cvpixelbuffer-cvimagebuffer-to-data – Daniel Apr 07 '20 at 16:18
  • No. Its a CMSampleBuffer – prabhu Apr 07 '20 at 16:40
  • I know its a CMSampleBuffer, but whats in it?? If its not a CMBlockBuffer that method wont work, it probably isnt a CMBlockBuffer, also take a look at https://developer.apple.com/documentation/coremedia/1489629-cmsamplebuffergetdatabuffer?language=objc – Daniel Apr 07 '20 at 16:42
  • Ok. Assume its not a CMBlockBuffer. How do i make it work to get CMBlockBuffer. What should i do with AVAcptureSession or the AVCaptureVideoDataOutput to return me CMBlockBuffer? – prabhu Apr 07 '20 at 16:55
  • Check whats in the CMSampleBuffer, like i said before, it could be CVImageBuffer and there is other ways to extract that data... or a CVPixelBuffer, look at the link i posted above of how to extract those – Daniel Apr 07 '20 at 17:00
  • you are correct. CMSampleBufferGetImageBuffer(sampleBuffer) will return CVImageBuffer. this works. not an issue. But what I need do after that is, I want to compress the CVImageBuffer. So I will pass this CVImageBuffer to VTCompressionSessionRef which after compression will return a CMSampleBuffer. From this I can not extract the compressed image data with CMSampleBufferGetImageBuffer as it is in compressed format. Hence I can only access with CMBlockBuffer which will return access to data buffer pointer. Since the block buffer is nil before compression, it is nil after compression also. – prabhu Apr 07 '20 at 17:14
  • I am new to this Screen capture functionality. I don't exactly know how this works. Since I used AVCaptureVideoDataOutput for output, I expect the sample buffer to contain CMBlockBuffer which will have accessed to the media data. – prabhu Apr 07 '20 at 17:16
  • There is a lot of documentation on this, just search... this might be what you are after https://developer.apple.com/documentation/accelerate/applying_vimage_operations_to_video_sample_buffers – Daniel Apr 07 '20 at 17:27
  • This is for iOS. I am searching for a solution in MacOS. I couldn't find any. That's why i posted this question – prabhu Apr 07 '20 at 17:42
  • IOS + MAcOS should be the same pretty much, they both have AVFoundation , have you actually tried it? There should be something similar if not for macOS.. https://developer.apple.com/av-foundation/ – Daniel Apr 09 '20 at 13:26

0 Answers0