Unexpected inputBuffer size when using mediacodec to encode yuv(NV_21) to mp4 file

Question

I tried to convert camera YUV data(NV21, got from onpreviewframe) to YUV420Semiplaner, and sent YUV420Semiplaner to mediacodec/mediamuxer for mp4 image. But I always found the size of inputBuffer was unexpected The following are my coding and logcat,

Mediacodec/Mediaformat configuration:

    MediaCodecInfo codecInfo = selectCodec(MIME_TYPE_VIDEO);
    int colorFormat = selectColorFormat(codecInfo, MIME_TYPE_VIDEO);

    formatVideo = MediaFormat.createVideoFormat(video/avc, 1280, 720);
    formatVideo.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
    formatVideo.setInteger(MediaFormat.KEY_BIT_RATE, 6000000);
    formatVideo.setInteger(MediaFormat.KEY_FRAME_RATE, 40);
    formatVideo.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);

    mVideoCodec = MediaCodec.createEncoderByType(video/avc);
    mVideoCodec.configure(formatVideo, null,null,MediaCodec.CONFIGURE_FLAG_ENCODE);

camera setting is below. I captured yuv from onpreviewframe and sent to codec.

    Camera.Parameters parameters = camera.getParameters();
    parameters.setPreviewFormat(ImageFormat.NV21);
    parameters.setPreviewSize(1280, 720);

    @Override
    public void onPreviewFrame(final byte[] sourceYUVData, Camera camera)
    {
    //sent sourceYUVData to Codec
        mVideoCodecInput.setByteBufferVideo(buffer, isUsingFrontCamera, Input_endOfStream);
    }

    //sent YUV data to Mediacodec dequeueinputbuffer
    public void setByteBufferVideo(byte[] buffer, boolean isUsingFrontCamera, boolean Input_endOfStream)
    {     
        ByteBuffer[] inputBuffers = mVideoCodec.getInputBuffers();
        videoInputBufferIndex = mVideoCodec.dequeueInputBuffer(-1); 

        if(videoInputBufferIndex>=0) {
            ByteBuffer inputBuffer = inputBuffers[videoInputBufferIndex];
            inputBuffer.clear();

            if (VERBOSE) {
                Log.w(TAG, put_v + "[put_v] buffer length = " + buffer.length + "; inputBuffer length =" + inputBuffer.capacity());
            }

            //skip following code
    }

I used 1280 x 720, and found when colorformat was YUV420semiplanar, the mediacodec dequeueinputbuffer size was 1425408, instead of 1382400 (1280x720x3/2). However when colorformat was YUV420Planar, the dequeue inputbuffer size is 1382400.

logcat:

    [put_v] buffer length = 1382400 ; inputBuffer length =1425408

Was the size correct or i got the wrong setting? If the size for YUV420Semiplanar was correct, could someone teach me how to convert NV21 to YUV420Semiplanar with the size? Thanks a lot.

There are some examples, e.g. the buffer-to-buffer tests in http://bigflake.com/mediacodec/#EncodeDecodeTest . You're generally better off working with Surfaces than raw YUV data. Do you have a requirement, e.g. 3rd party library, that requires you to use the YUV buffer? Have you tried just copying the data into the top of the buffer to see what results you get? — fadden, Mar 17 '16 at 20:01
Yes, I already had other application that needed the yuv data from camera preview, and change to surface and shader will introduce a huge change on my current code flow. I had tried to copy the yuv into the dequeue inputbuffer regardless the buffer size mismatch, i found the video will be discontinuous when using mediaplayer to play the output mp4 file. Thanks for the response. — Yu-Cheng Fan, Mar 18 '16 at 02:48
Not sure what "discontinuous" means. What version of Android are you using? Have you tried more than once device? This sounds like buffer alignment issues, but I wouldn't expect that to happen on API 18+. (1425408 is 16384 * 87, so it could be aligning the different planes to 16K boundaries. The math isn't *quite* right though.) — fadden, Mar 18 '16 at 04:29
I meant the video was not smooth or a bit lag. I observed the problem on LG Nexus 6P with android 6.0.1 (API 23) and HTC m9 with android 5 (API 21). However, there was no problem on Samsung Galaxy S6 and note4 with android (API 21). The difference i observed between success and failure cases are that Samsung devices used YUV420Planar (no problem, dequeue inputbuffer size = 1382400), but Nexus and HTC used YUV420SemiPlanar(failure, buffer size 1425408). Would the mismatch buffer sizes be a problem, or I can just ignore the difference and look into the other problem? — Yu-Cheng Fan, Mar 18 '16 at 06:13
If the buffer contents didn't match the color format you'd see weird color effects. The pacing of the frames is not related. Make sure the presentation timestamps you're passing through have the correct interval and units (i.e. microseconds). — fadden, Mar 18 '16 at 16:26

score 0 · Answer 1 · edited May 23 '17 at 12:31

Thanks Fadden! By your suggestion, I already clarified the mismatch buffer size won't lead to pacing of frames. I found the pacing problem was gone when i commented the rotation function. Now I am trying to optimize the code flow. For the rotation function, i used Rotate YUV420/NV21 Image in android

    byte[] bufferOutputRotation = new byte[buffer.length];
    public void setByteBufferVideo(byte[] buffer, boolean isUsingFrontCamera, boolean Input_endOfStream){
    if(Build.VERSION.SDK_INT >=18){
        try{

            endOfStream = Input_endOfStream;
            if(!Input_endOfStream){
            ByteBuffer[] inputBuffers = mVideoCodec.getInputBuffers();
            videoInputBufferIndex = mVideoCodec.dequeueInputBuffer(-1);

                if (VERBOSE) {
                    Log.w(TAG,"[put_v]:"+(put_v)+"; videoInputBufferIndex = "+videoInputBufferIndex+"; endOfStream = "+endOfStream);
                }

                if(videoInputBufferIndex>=0) {
                    ByteBuffer inputBuffer = inputBuffers[videoInputBufferIndex];
                    inputBuffer.clear();

                    if(isUsingFrontCamera){
                        bufferOutputRotation=buffer;
                    }else {
                        bufferOutputRotation=buffer;
                       //issue was gone by comment the rotation for rear camera
                       //rotateNV21(buffer,bufferOutputRotation,1280,720,180);
                    }

                    inputBuffer.put(mNV21Convertor.convert(bufferOutputRotation));
                    videoInputLength = buffer.length;

Unexpected inputBuffer size when using mediacodec to encode yuv(NV_21) to mp4 file

1 Answers1