We’re attempting to save video and audio from an Android device into an encrypted file. Our current implementation pipes the outputs from microphone and camera through the MediaEncoder class. As the data is output from MediaEncoder, we are encrypting and writing the contents of the byte buffer to disk. This approach works, however, when attempting to stitch the files back together with FFMPEG, we notice that the two streams seem to get out of sync somewhere mid stream. It appears that a lot of important metadata is lost with this method, specifically presentation timestamps and frame rate data as ffmpeg has to do some guess work to mux the files.
Are there techniques for keeping these streams in sync without using MediaMuxer? The video is encoded with H.264 and the audio with AAC.
Other Approaches: We attempted to use the MediaMuxer to mux the output data to a file, but our use case requires that we encrypt the bytes of data before they are saved to disk which eliminates the possibility of using the default constructor.
Additionally, we have attempted to use the newly added (API 26) constructor that takes a FileDescriptor instead and have that pointed to a ParcelFileDescriptor that wrapped an Encrypted Document (https://android.googlesource.com/platform/development/+/master/samples/Vault/src/com/example/android/vault/EncryptedDocument.java). However, this approach led to crashes at the native layer and we believe it may have to do with this comment from the source code (https://android.googlesource.com/platform/frameworks/base.git/+/master/media/java/android/media/MediaMuxer.java#353) about the native writer trying to memory map the output file.
import android.graphics.YuvImage
import android.media.MediaCodec
import android.media.MediaCodecInfo
import android.media.MediaFormat
import android.media.MediaMuxer
import com.callyo.video_10_21.Utils.YuvImageUtils.convertNV21toYUV420Planar
import java.io.FileDescriptor
import java.util.*
import java.util.concurrent.atomic.AtomicReference
import kotlin.properties.Delegates
class VideoEncoderProcessor(
private val fileDescriptor: FileDescriptor,
private val width: Int,
private val height: Int,
private val frameRate: Int
): MediaCodec.Callback() {
private lateinit var videoFormat: MediaFormat
private var trackIndex by Delegates.notNull<Int>()
private var mediaMuxer: MediaMuxer
private val mediaCodec = createEncoder()
private val pendingVideoEncoderInputBufferIndices = AtomicReference<LinkedList<Int>>(LinkedList())
companion object {
private const val VIDEO_FORMAT = "video/avc"
}
init {
mediaMuxer = MediaMuxer(fileDescriptor, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
mediaCodec.setCallback(this)
mediaCodec.start()
}
private fun createEncoder(): MediaCodec {
videoFormat = MediaFormat.createVideoFormat(VIDEO_FORMAT, width, height).apply {
setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible)
setInteger(MediaFormat.KEY_BIT_RATE, width * height * 5)
setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
}
return MediaCodec.createEncoderByType(VIDEO_FORMAT).apply {
configure(videoFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
}
}
override fun onInputBufferAvailable(codec: MediaCodec, index: Int) {
// logic for handling stream end omitted for clarity
/* Video frames come in asynchronously from input buffer availability
* so we need to keep track of available buffers in queue */
pendingVideoEncoderInputBufferIndices.get().add(index)
}
override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {}
override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) {
trackIndex = mediaMuxer.addTrack(format)
mediaMuxer.start()
}
override fun onOutputBufferAvailable(codec: MediaCodec, index: Int, bufferInfo: MediaCodec.BufferInfo) {
val buffer = mediaCodec.getOutputBuffer(index)
buffer?.apply {
if (bufferInfo.size != 0) {
limit(bufferInfo.offset + bufferInfo.size)
rewind()
mediaMuxer.writeSampleData(trackIndex, this, bufferInfo)
}
}
mediaCodec.releaseOutputBuffer(index, false)
if (bufferInfo.flags == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
mediaCodec.stop()
mediaCodec.release()
mediaMuxer.stop()
mediaMuxer.release()
}
}
// Public method that receives raw unencoded video data
fun encode(yuvImage: YuvImage) {
// logic for handling stream end omitted for clarity
pendingVideoEncoderInputBufferIndices.get().poll()?.let { index ->
val buffer = mediaCodec.getInputBuffer(index)
buffer?.clear()
// converting frame to correct color format
val input =
yuvImage.convertNV21toYUV420Planar(ByteArray(yuvImage.yuvData.size), yuvImage.width, yuvImage.height)
buffer?.put(input)
buffer?.let {
mediaCodec.queueInputBuffer(index, 0, input.size, System.nanoTime() / 1000, 0)
}
}
}
}
Additional Info: I’m using MediaCodec.Callback() (https://developer.android.com/reference/kotlin/android/media/MediaCodec.Callback?hl=en) to handle the encoding asynchronously.
