MediaCodec Output buffer doesn't generate correct output when it's input surface is mirrored to the AR sceneview surface

Question

I am currently working on a project which requires the contents(the camera image along with the AR objects for each frame) of the SceneView in a byte array format to stream the data.

I have tried to Mirror the SceneView to a MediaCodec encoder's input surface and use the MediaCodec's output buffer in an asynchronous manner based on what I understood from the MediaRecorder sample.

I have been unable to get it to work as I expect it to. The output buffer from the MediaCodec's callback either displays a black screen (when converted to bitmap) or it has way too little buffer contents (<100 bytes). I have a feeling that the mirroring to a surface is not occurring as I expect it to, It would be much appreciated if someone could provide a sample that showcases the proper way to use MediaCodec with SceneForm

The code I am currently using for accessing the ByteBuffer from MediaCodec :

MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC,
        arFragment.getArSceneView().getWidth(),
        arFragment.getArSceneView().getHeight());

// Set some properties to prevent configure() from throwing an unhelpful exception.
format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
        MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
format.setInteger(MediaFormat.KEY_BIT_RATE, 50000);
format.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 0); //All key frame stream

MediaCodec mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);

mediaCodec.setCallback(new MediaCodec.Callback() {
    @Override
    public void onInputBufferAvailable(@NonNull MediaCodec codec, int index) {
    }

    @Override
    public void onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo info) {
        ByteBuffer outputBuffer = mediaCodec.getOutputBuffer(index);
        outputBuffer.position(info.offset);
        outputBuffer.limit(info.offset + info.size);
        byte[] data = new byte[outputBuffer.remaining()];
        outputBuffer.get(data);
        Bitmap bitmap = BitmapFactory.decodeByteArray(
                NV21toJPEG(data, videoWidth, videoHeight, 100),
                0, data.length);
        Log.d(TAG, "onOutputBufferAvailable: "+bitmap);
        mediaCodec.releaseOutputBuffer(index, false);
    }

    @Override
    public void onError(@NonNull MediaCodec codec, @NonNull MediaCodec.CodecException e) {
        Log.e(TAG, "onError: ");
    }

    @Override
    public void onOutputFormatChanged(@NonNull MediaCodec codec, @NonNull MediaFormat format) 
   {
        Log.d(TAG, "onOutputFormatChanged: " + format);
    }
});

mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
Surface surface = mediaCodec.createInputSurface();
mediaCodec.start();
arFragment.getArSceneView().startMirroringToSurface(surface, 0, 0, arFragment.getArSceneView().getWidth(), arFragment.getArSceneView().getHeight());

The generated bitmap :

Generated bitmap

score 0 · Answer 1 · answered Feb 13 '19 at 12:17

0

The input to the MediaCodec encoder is a Surface (in your case, could be a byte buffer as well) with raw YUV data (not directly accessible). The output from the encoder is an encoded H264/AVC bitstream.

In your case, it seems like you are trying to read the encoded bitstream and interpret it as raw YUV data.

Do you really want to encode the input data into a video format (for streaming?), or are you just trying to use MediaCodec to convert data from a Surface into YUV data accessible in a byte buffer? You could get a hand on the YUV data creating a MediaCodec decoder and promptly feeding data from the encoder to the decoder, but that's all very roundabout.

The direct way of receiving data through a Surface and accessing the pixel data is via the ImageReader class. See Using Camera2 API with ImageReader for a bigger example of using this.

answered Feb 13 '19 at 12:17

mstorsjo

12,983
2
39
62

Thanks for the quick reply, my intention for using MediaCodec is to get the YUV data from the Ar SceneView's surface for video streaming. I have tried to use the ImageReader class in a similar manner by mirroring the SceneView's surface to it and acquiring the latest image when it is available (using setOnImageAvailableListener), but it's performance seems very poor. The onImageAvailable seems to be called once in every 100-300 ms, which makes it practically useless for live streaming. – Chrisvin Jem Feb 13 '19 at 13:54
I assumed that the poor performance was because ImageReader waits for the Surface to first render it's contents before reading it (as bytes) and generating the image. Therefore since MediaCodec directly provided the bytes, I was under the impression that it would be faster in general. Is there anything wrong with my understanding of how ImageReader and MediaCodec works? – Chrisvin Jem Feb 13 '19 at 13:59
You seem to be overlooking the fact that MediaCodec isn't meant as a way to access YUV data, but for actually doing lossy encoding of a video stream. MediaCodec and ImageReader aren't two alternative approaches for doing the same thing. ImageReader is for reading raw pixel data out of a Surface, while MediaCodec is for encoding and decoding. MediaCodec doesn't "directly provide the bytes", it also does a costly encoding (although it in general should be able to keep up with full framerate). You shouldn't be doing an encoding in vain just as an API adapter. – mstorsjo Feb 13 '19 at 14:22
I have done tests with ImageReader in other situations, where it seemed to work just fine for me. Not sure if there are additional bottlenecks in your case. When testing with ImageReader, have you tested with an empty callback that just does `reader.acquireNextImage().close();` or something similar, to see what the best case performance is when you do nothing? – mstorsjo Feb 13 '19 at 14:22
I need the data in a YUV format, hence the encoding that MediaCodec did seemed to suit my requirements. But I understand how it would be bad practice to do so in general. I did a quick test based on your suggestion with the following code, ~~~ public void onImageAvailable(ImageReader reader) { reader.acquireNextImage().close(); Log.d(TAG, "onImageAvailable: "+System.currentTimeMillis()); } ~~~ And the average time between each onImageAvailable call seemed to be around 50-150 ms. – Chrisvin Jem Feb 13 '19 at 15:01
I did a bit of searching and this seems to be pretty common issue, [link 1](https://stackoverflow.com/questions/49872612/why-camera2-use-imagereader-get-yuv-image-for-process-just-only-under-10-15-fra) [link 2](https://stackoverflow.com/questions/51006362/why-is-androids-imagereader-class-so-slow) [link 3](https://stackoverflow.com/questions/42688188/android-camera2-output-to-imagereader-format-yuv-420-888-still-slow) . If you can, could you post your code for the ImageReader in one of those issues, it would be quite useful. – Chrisvin Jem Feb 13 '19 at 15:06
And back to my problem, I would appreciate it if a solution using MediaCodec was provided, since the Sceneform documentation states that it is possible to use MediaCodec but I have been unable to find any sample code for such usage. This would serve as a useful future reference and I suspect that usage of MediaCodec would be more suited for my current requirements. – Chrisvin Jem Feb 13 '19 at 15:08
Well MediaCodec would suit you if you want to store the output in a video file or stream it over the network, but it's definitely the wrong match for reading out YUV data from a surface. An encode-decode pipeline might work though, and it would probably give decent performance even (but with a couple frames of latency), but that'd be a huge kludge. – mstorsjo Feb 13 '19 at 15:56
As for ImageReader performance, I guess that varies a lot between devices and between what produces the input. I don't have that test code readily available at the moment unfortunately. – mstorsjo Feb 13 '19 at 15:58

MediaCodec Output buffer doesn't generate correct output when it's input surface is mirrored to the AR sceneview surface

1 Answers1