1

My app recognises an event using Vision and uses CMSampleBuffer to do so. After the event I am recording the video already using AVWriter successfully.

Now I want to record the full motion and thus record 1-2 seconds before the event occurred.

I tried pushing the CMSampleBuffer into a ring buffer, but that starves the camera of buffers.

func captureOutput(_ output: AVCaptureOutput,
                       didOutput sampleBuffer: CMSampleBuffer,
                       from connection: AVCaptureConnection) {
// sends that to detectBall

  /// Gets called by the camera every time there is a new buffer available
 func detectBall(inBuffer buffer: CMSampleBuffer,
                      ballDetectionRequest: VNCoreMLRequest,
                      orientation: CGImagePropertyOrientation,
                      frame: NormalizedPoint,
                      updatingRingBuffer: PassthroughSubject<AppEnvironment.AVState.RingBufferItem, Never>
  ) throws {
    // I tried to convert it into a CVPixelBuffer but its a shallow copy as well so it also starves the camera
    let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(buffer)!
    /// rotated 90 because of the cameras native landscape orientation
   
    
    let visionHandler = VNImageRequestHandler(ciImage: croppedImage, options: [:])
    try visionHandler.perform([ballDetectionRequest])
    if let results = ballDetectionRequest as? [VNClassificationObservation] {
      // Filter out classification results with low confidence
      let filteredResults = results.filter { $0.confidence > 0.9 }
      guard let topResult = results.first,
            topResult.confidence > 0.9 else { return }
      // print("       its a: \(topResult.identifier)")
      // print("copy buffer")
      
      updatingRingBuffer.send(AppEnvironment.AVState.RingBufferItem(
    /// HERE IS THE PROBLEM: AS SOON AS I SEND IT SOMEWHERE ELSE THE CAMERA IS STARVED
                                buffer: imageBuffer,
                                ball: topResult.identifier == "ball")

How can I achieve to store these 1-2 seconds of video continuously without writing it to disk and then prepending it to the video file?

Thanks!

bnassler
  • 591
  • 6
  • 15
  • You could duplicate the `CMSampleBuffer`s, but even 2 seconds can require hundreds of megabytes of memory. You could compress the captured video to a file or even to memory, using the new fragmented mp4 capabilities of `AVAssetWriter`, like this solution does: https://stackoverflow.com/a/66829118/22147 – Rhythmic Fistman Apr 15 '21 at 06:53
  • @RhythmicFistman Thanks so much for your reply, I got the duplicate to work somewhat by converting them to CVPixelBuffer but then the timing is off and I couldn't figure out why. But you're completely right there is a huge potential issue with RAM. Thanks a million for the link I know about the fragmented mp4 but thought it would write to disk. Let me check it out and come back to you! That would be amazing – bnassler Apr 15 '21 at 16:36

0 Answers0