Android: How to use MediaMuxer with video/mp4v-es instead of video/avc?

Question

I want to be able to use mp4v-es instead of avc on some devices. The encoder runs fine using avc, but when I replace it with mp4v-es, the muxer reports:

E/MPEG4Writer(12517): Missing codec specific data

as in MediaMuxer error "Failed to stop the muxer", and the video cannot be played. The difference is that I am adding the correct track/format to the muxer, without receiving any error:

...else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
           MediaFormat newFormat = encoder.getOutputFormat();
           mTrackIndex[encID] = mMuxer.addTrack(newFormat);

Is there any difference in handling mp4v-es compared to avc? One mention, I just skip "bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG" when it occurs, as for avc it was not needed.Thanks.

Answer 1

I presume you have the ability to modify the Stagefright sources and hence, I have a proposed solution for your problem, but one which requires a customization.

Background:

When an encoder completes encoding, the first buffer will have the csd information which is usually tagged with OMX_BUFFERFLAG_CODECCONFIG flag. When such a buffer is returned to the MediaCodec, it shall store the same as csd-0 in MediaCodec::amendOutputFormatWithCodecSpecificData.

Now, when this buffer is given to MediaMuxer, the same is processed as part of addTrack,in which convertMessageToMetadata is invoked. If you refer to the implementation of the same, we can observe that only AVC is handled for video and defaults to audio for ESDS creation.

EDIT:

Here, my recommendation is to modify this line as below and try your experiment

} 
if (mime.startsWith("audio/") || (!strcmp(mime, MEDIA_MIMETYPE_VIDEO_MPEG4)) {

With this change, I feel it should work for MPEG4 video track also. The change is to convert the else if into if as the previous check for video will also try to process the data, but only for AVC.

Answer 2

Just as Ganesh pointed out, unfortunately it does seem that this isn't possible right now, without modifying the platform source.

There's actually two ways that the codec specific data can be passed to the internal MPEG4Writer class, but neither of them actually work without modifications.

As Ganesh found, the logic for remapping MediaFormat keys to the internal format seems to be missing handling of codec specific data for any other video codec than H264. A tested modification that fixes this issue is as follows:

diff --git a/media/libstagefright/Utils.cpp b/media/libstagefright/Utils.cpp
index 25afc5b..304fe59 100644
--- a/media/libstagefright/Utils.cpp
+++ b/media/libstagefright/Utils.cpp
@@ -549,14 +549,14 @@ void convertMessageToMetaData(const sp<AMessage> &msg, sp<MetaData> &meta) {
     // reassemble the csd data into its original form
     sp<ABuffer> csd0;
     if (msg->findBuffer("csd-0", &csd0)) {
-        if (mime.startsWith("video/")) { // do we need to be stricter than this?
+        if (mime == MEDIA_MIMETYPE_VIDEO_AVC) {
             sp<ABuffer> csd1;
             if (msg->findBuffer("csd-1", &csd1)) {
                 char avcc[1024]; // that oughta be enough, right?
                 size_t outsize = reassembleAVCC(csd0, csd1, avcc);
                 meta->setData(kKeyAVCC, kKeyAVCC, avcc, outsize);
             }
-        } else if (mime.startsWith("audio/")) {
+        } else if (mime == MEDIA_MIMETYPE_AUDIO_AAC || mime == MEDIA_MIMETYPE_VIDEO_MPEG4) {
             int csd0size = csd0->size();
             char esds[csd0size + 31];
             reassembleESDS(csd0, esds);

Secondly, instead of passing the codec specific data as csd-0 in MediaFormat, you could in principle pass the same buffer (with the MediaCodec.BUFFER_FLAG_CODEC_CONFIG flag set) to MediaMuxer.writeSampleData. This approach doesn't work currently since this method doesn't check for the codec config flag at all - it could be fixed with this modification:

diff --git a/media/libstagefright/MediaMuxer.cpp b/media/libstagefright/MediaMuxer.cpp
index c7c6f34..d612e01 100644
--- a/media/libstagefright/MediaMuxer.cpp
+++ b/media/libstagefright/MediaMuxer.cpp
@@ -193,6 +193,9 @@ status_t MediaMuxer::writeSampleData(const sp<ABuffer> &buffer, size_t trackInde
     if (flags & MediaCodec::BUFFER_FLAG_SYNCFRAME) {
         sampleMetaData->setInt32(kKeyIsSyncFrame, true);
     }
+    if (flags & MediaCodec::BUFFER_FLAG_CODECCONFIG) {
+        sampleMetaData->setInt32(kKeyIsCodecConfig, true);
+    }

     sp<MediaAdapter> currentTrack = mTrackList[trackIndex];
     // This pushBuffer will wait until the mediaBuffer is consumed.

As far as I can see, there's no way to mux MPEG4 video with MediaMuxer right now while using the public API, without modifying the platform source. Given the issues in Utils.cpp above, you can't mux any video format that requires codec specific data, except for H264. If VP8 is an option, you can mux that into webm files (together with vorbis audio), but hardware encoders for VP8 is probably much less common than hardware encoders for MPEG4.