2

Background

I use AVAssetWriterInput.append to append sample buffers to the writer. Sometimes, I switch off the audio input (if the user wants to temporarily disable audio input), so the append method will not be executed while the append method in the video input will always be executed.

Problem

If the user pauses the audio and resumes it later, the audio after resuming will immediately begin when the user pauses it (in the final video).

Example

'=' refers to CMSampleBuffer.

'|' means the user paused the audio input.

Video: ---------------=================================

Audio(expected): ----=======|----------------=============

Audio(I got): ---------=======|=============----------------

Code

func appendBuffer(_ buffer: CMSampleBuffer, of type: BufferType) {
    guard let writer else { return }
    guard writer.status == .writing else {
        logger.warning("AVAssetWriter is not ready. Status: \(writer.status.rawValue). Error: \(writer.error.debugDescription)")
        return
    }
    
    // Start a session when receives the first frame.
    if isFirstFrame && type == .screenVideo {
        startInputPipeline(with: buffer)
        isFirstFrame = false
    }
    
    guard isWriting else { return }
    
    switch type {
    case .video:
        // Check the status of the buffer to decide whether to append it or not.
        guard statusOfSampleBuffer(buffer) == .complete else { return }
        if videoInput?.isReadyForMoreMediaData == true {
            guard buffer.imageBuffer != nil else {
                logger.info("Complete but no updated pixels.")
                return
            }
            processQueue.async { [self] in
                videoInput?.append(buffer)
            }
        }
    case .audio:
        if audioInput?.isReadyForMoreMediaData == true {
            guard buffer.dataBuffer != nil else { return }
            processQueue.async { [self] in
                audioInput?.append(buffer)
            }
        }
    }
}

I have printed the presentationTime from the audio sample buffer. It turns out it's correct.

Maybe my understanding of AVAssetWriterInput.append is wrong?

My current solution is to always append the buffer, but when the user wants to pause, I simply append an empty SampleBuffer filled with nothing.

I don't think this is the best way to deal with it.

Is there any way to sync the buffer time with the video?

LiYanan2004
  • 41
  • 5
  • Could you please share how you append your buffers? – Bulat Yakupov Oct 30 '22 at 18:15
  • @BulatYakupov I have added the code in the question. Please check it out. – LiYanan2004 Oct 31 '22 at 01:37
  • @BulatYakupov When the audio is paused, the delegate method will not give me any audio buffer until I resume it. After resuming, the buffer will be appended **just after the last buffer** when I paused the audio. (You can check out the example above.) – LiYanan2004 Oct 31 '22 at 01:48
  • 1
    Ok, I made a research and it seems that your current solution is the correct one. AVAssetWriterInput ignores pauses and concats buffers. So you need to fill this pause with something. – Bulat Yakupov Nov 03 '22 at 11:03

1 Answers1

3

I would recommend stepping away form pausing the writing and just write silent audio buffers or black video frames It's easy to do

For Audio

    private func _cacheAudioBuffer(sampleBuffer: CMSampleBuffer, isMuted: Bool) {
    if isMuted,
       let ref = CMSampleBufferGetDataBuffer(sampleBuffer) {
        CMBlockBufferFillDataBytes(
            with: 0,
            blockBuffer: ref,
            offsetIntoDestination: 0,
            dataLength: CMBlockBufferGetDataLength(ref))
    }
    // Continue here with muted samplebuffer
}

For Video (you can play with the frame properties to fit your needs)

    private func _createBlackPixelBuffer(with image: UIImage?, with size: PresetDimension) -> CVPixelBuffer? {
    guard self._blackPixelBuffer == nil else { return self._blackPixelBuffer }
    var pixelBuffer: CVPixelBuffer?
    let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
                 kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
    let width = Int(size.width)
    let height = Int(size.height)
    CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, attrs, &pixelBuffer)
    let context = CIContext()
    guard let image = image,
        let ciImage = CIImage(image: image) else {
        return nil
    }
    context.render(ciImage, to: pixelBuffer!)
    return pixelBuffer
}

Or if you wish to stop the file when you pause, I would stop the writing and start a new one when the user starts it again

YYfim
  • 1,402
  • 1
  • 9
  • 24