How can I tell what MediaFormats are supported for video encoding on an Android device?

Question

I am developing a library for video/photo processing (add filters like Instagram/Snapchat). So far the core features work very well.

However, I am finding the processing of a video (re-encoding an input video) to be hugely frustrating. There seem to be a number of edge cases and device specific issues that prevent the library from working 100% of the time.

I would like to know how to select/create a MediaFormat that will work on a device.

Currently, I'm setting up the MediaFormat that will be used to encode a video as follows:

// assume that "extractor" is a media extractor wrapper, which holds a 
// reference to the MediaFormat of the input video
fun getOutputVideoFormat(): MediaFormat {
    val mimeType = MediaFormat.MIMETYPE_VIDEO_H263
    var width = -1
    var height = -1
    var frameRate = 30
    var bitrate = 10_000_000
    val colorFormat = MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface

    if (extractor.videoFormat.containsKey(MediaFormat.KEY_WIDTH)) {
        width = extractor.videoFormat.getInteger(MediaFormat.KEY_WIDTH)
    }

    if (extractor.videoFormat.containsKey(MediaFormat.KEY_HEIGHT)) {
        height = extractor.videoFormat.getInteger(MediaFormat.KEY_HEIGHT)
    }

    if(extractor.videoFormat.containsKey(MediaFormat.KEY_FRAME_RATE)){
        frameRate = extractor.videoFormat.getInteger(MediaFormat.KEY_FRAME_RATE)
    }

    if(extractor.videoFormat.containsKey(MediaFormat.KEY_BIT_RATE)){
        bitrate = extractor.videoFormat.getInteger(MediaFormat.KEY_BIT_RATE)
    }

    val format = MediaFormat.createVideoFormat(mimeType, width, height)
    format.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat)
    format.setInteger(MediaFormat.KEY_BIT_RATE, bitrate)
    format.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
    format.setInteger(MediaFormat.KEY_CAPTURE_RATE, frameRate)

    // prevent crash on some Samsung devices
    // http://stackoverflow.com/questions/21284874/illegal-state-exception-when-calling-mediacodec-configure?answertab=votes#tab-top
    format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, width * height)
    format.setInteger(MediaFormat.KEY_MAX_WIDTH, width)
    format.setInteger(MediaFormat.KEY_MAX_HEIGHT, height)
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 0)
    return format
}

So far this works on all major devices that I have tested with, but there are some devices like the Samsung A5 that have been reported to fail silently using this format, and simply create corrupted output videos using an input video that works correctly on all other devices.

How can I tell if a MediaFormat will actually succeed on a given device?

The only logs I have from the Samsung A5 device indicate that when the MediaCodec sends through the "INFO_OUTPUT_FORMAT_CHANGED" signal, the following media format is returned:

csd-1=java.nio.ByteArrayBuffer[position=0,limit=8,capacity=8], 
mime=video/avc,  
frame-rate=30,  
remained_resource=2549760,  
height=480,  
width=480,  
max_capacity=3010560, what=1869968451,  
bitrate=10000000,  
csd-0=java.nio.ByteArrayBuffer[position=0,limit=17,capacity=17]

This format seems invalid to me, given the fact the input video has a resolution of 1280x720

mstorsjo · Answer 1 · 2018-03-22T08:34:50.737

2

You can use the MediaCodecList API to query and list what codecs are available and what formats they support.

Also for your code example, are you really using MediaFormat.MIMETYPE_VIDEO_H263 or is that a typo? That's a very old format. (Not old in the "well supported and reliable" way but in the "old, untested and possibly broken" way.) The safest thing would be to use MediaFormat.MIMETYPE_VIDEO_AVC which is the one that gets the most testing, both by the Android compatibility testsuite and by third party applications.

edited Mar 22 '18 at 08:34

answered Mar 21 '18 at 07:19

mstorsjo

12,983
2
39
62

I'm using MediaFormat.MIMETYPE_VIDEO_AVC (which is what I assume you mean by MIMETYPE_VIDEO_H264?) on the main branch for the format, but I switched to MIMETYPE_VIDEO_H263 to test against this issue as I assumed the older codec would have better support. – isaac.udy Mar 21 '18 at 23:39
Checking out the MediaCodecList solution now., Thanks for pointing me towards that - I'm quite surprised that I hadn't found it already through the hours of Googling that I've done! – isaac.udy Mar 21 '18 at 23:40
Yes, I meant `MediaFormat.MIMETYPE_VIDEO_AVC` - sorry for the confusion. In general, older formats may be better supported, but I wouldn't think so in this case; especially as the H263 bitstream format actually is limited to a fixed number of low resolutions. – mstorsjo Mar 22 '18 at 08:36
Thanks for your response, but apparently my issue actually has nothing to do with video codecs. The actual problem was to do with *audio*. I have posted a full answer above if you'd like to understand what I was doing wrong. – isaac.udy Mar 23 '18 at 09:11

score 0 · Accepted Answer · answered Mar 23 '18 at 09:35

It turns out that my problem had nothing to do with the video codecs available on the device. The problem didn't come from MediaCodec or MediaFormat, but from MediaMuxer.

I was processing video and audio by reading them out through a MediaExtractor, pushing that through a MediaCodec configured for decoding, processing that data, and then pushing processed data through a MediaCodec configured for encoding. Then I was pushing the encoded data to a MediaMuxer (and eventually writing it to a file). This is very similar to the DecodeEditEncodeTest found on https://bigflake.com/mediacodec/. I was only doing processing on the video track, but I was using a similar decode/encode method to take the audio from the input file and put it into the output file.

I initially thought the problem was device specific, but it turns out that the issue was actually with the input. The video that was causing the issue with processing was very short - less than 2 seconds long. Decoding and re-encoding the audio was not working correctly with such a short video, and the MediaMuxer was not registering any audio frames. This is what was causing the final output to be corrupted.

I found the following CTS test: https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/MediaMuxerTest.java, and its cloneMediaUsingMuxer method, which shows how to copy audio directly from a MediaExtractor into a MediaMuxer.

I modified my processing method to (continue to) use the decode/edit/encode method for video, and to use the passthrough method demonstrated by the CTS test for writing audio. This solved the issue, and I was able to process the short video correctly.

How can I tell what MediaFormats are supported for video encoding on an Android device?

2 Answers2