4

I'd like to display decoded video frames from MediaCodec out of order, or omit frames, or show frames multiple times.

I considered configuring MediaCodec to use a Surface, call MediaCodec.dequeueOutputBuffer() repeatedly, save the resulting buffer indices, and then later call MediaCodec.releaseOutputBuffer(desired_index, true), but there doesn't seem to be a way to increase the number of output buffers, so I might run out of output buffers if I'm dealing with a lot of frames to be rearranged.

One idea I'm considering is to use glReadPixels() to read the pixel data into a frame buffer, convert the color format appropriately, then copy it to a SurfaceView when I need the frame displayed. But this seems like a lot of copying (and color format conversion) overhead, especially when I don't inherently need to modify the pixel data.

So I'm wondering if there is a better, more performant way. Perhaps there is a way to configure a different Surface/Texture/Buffer for each decoded frame, and then a way to tell the SurfaceView to display a specific Surface/Texture/Buffer (without having to do a memory copy). It seems like there must be a way to accomplish this with OpenGL, but I'm very new to OpenGL and could use recommendations on areas to investigate. I'll even go NDK if I have to.

So far I've been reviewing the Android docs, and fadden's bigflake and Grafika. Thanks.

Xargs
  • 769
  • 8
  • 12

1 Answers1

3

Saving copies of lots of frames could pose a problem when working with higher-resolution videos and higher frame counts. A 1280x720 frame, saved in RGBA, will be 1280x720x4 = 3.5MB. If you're trying to save 100 frames, that's 1/3rd of the memory on a 1GB device.

If you do want to go this approach, I think what you want to do is attach a series of textures to an FBO and render to them to store the pixels. Then you can just render from the texture when it's time to draw. Sample code for FBO rendering exists in Grafika (it's one of the approaches used in the screen recording activity).

Another approach is to seek around in the decoded stream. You need to seek to the nearest sync frame before the frame of interest (either by asking MediaExtractor to do it, or by saving off encoded data with the BufferInfo flags) and decode until you reach the target frame. How fast this is depends on how many frames you need to traverse, the resolution of the frames, and the speed of the decoder on your device. (As you might expect, stepping forward is easier than stepping backward. You may have noticed a similar phenomena in other video players you've used.)

Don't bother with glReadPixels(). Generally speaking, if decoded data is directly accessible from your app, you're going to take a speed hit (more so on some devices than others). Also, the number of buffers used by the MediaCodec decoder is somewhat device-dependent, so I wouldn't count on having more than 4 or 5.

fadden
  • 51,356
  • 5
  • 116
  • 166
  • Thanks @fadden. Your FBO approach makes sense. It will take me some time to understand all the APIs, RecordFBOActivity.java, and how to get data from MediaCodec to render to the FBO. (It seems like the idea is to do like CameraSurfaceRenderer and create a texture, associated SurfaceTexture, and associated Surface, which is passed to MediaCodec.configure(), and then when SurfaceTexture.OnFrameAvailableListener.onFrameAvailable() is called, call updateTexImage() to update the texture, then switch to the FBO, switch to a texture, and draw the original texture into the current texture.) – Xargs Jan 28 '14 at 03:08
  • That flow sounds right. You're essentially copying a GL texture to another GL texture. The "easy" way to do this would be to attach the SurfaceTexture's texture to an FBO, and then use `glCopyTexImage2D()` to read from the FBO and write to a new texture, but that doesn't work (http://stackoverflow.com/questions/19366660/how-to-save-surfacetexture-as-bitmap#19370209). The SurfaceTexture API doesn't provide a way to "swap out" the texture (the "detach" call deletes the texture). The good news is that it should be a fast blit under the covers (+/- a YUV -> RGB conversion). – fadden Jan 28 '14 at 06:24
  • Thanks, glad to hear that I'm not totally off-base. BTW, regarding the memory: it is a concern, but I think it will be manageable for my application. For comparison, I sometimes see Chrome grow to use 1/4th of the memory on a 1GB device. As to seeking the stream, the performance might not be enough for my application. I modified Grafika’s MoviePlayer class to ignore frameCallback.preRender(), so that it would render as fast as possible. Playing back a 9Mbps 1080p 21fps file was about 43fps on a Nexus 7 (2013). Pretty good, worth a prototype, but maybe not enough slack. – Xargs Jan 28 '14 at 08:02
  • The performance should improve a bit if you also disable rendering for the frames you're skipping over. I think there will be a lot of variation between devices, but that's going to be true no matter what approach you use. – fadden Jan 28 '14 at 15:41
  • I got a prototype of the FBO approach working and the performance is very good. I tried the same code on a Galaxy Nexus (4.2.2) and MediaCodec.dequeueOutputBuffer() kept returning BufferInfo.size==0. Will have to debug that further (maybe I'll try some of your 4.3 CTS tests on the device), but just wanted to say thanks. – Xargs Feb 05 '14 at 03:04
  • Getting a size of zero is normal when decoding to a Surface, because the data isn't actually put into the buffer that you have access to. – fadden Feb 05 '14 at 05:18
  • But then why does Grafika only pass render=true to releaseOutputBuffer() if BufferInfo.size!=0? (Line 245 of MoviePlayer.java) – Xargs Feb 05 '14 at 05:46
  • I tried ExtractMpegFramesTest.java on a Galaxy Nexus (4.2.2) and BufferInfo.size is always 0. This causes the code to use releaseOutputBuffer(doRender=false), which prevents onFrameAvailable() from ever being called, preventing the sample from working. Q: is the info.size!=0 check in Grafika/ExtractMpegFramesTest the wrong check to make? Should doRender always be set to true? Or is the code correct, but this is just considered a problem with an old device / old platform (which never had decode-to-surface CTS tests)? Thx – Xargs Feb 05 '14 at 09:50
  • Arrrgh, I confused myself. :-) A size of zero is *not* normal; a null buffer reference is normal. The basic CTS tests (notably EncodeDecodeTest) do expect a nonzero size, so any device running 4.3+ will behave as expected. I'm not sure what the 4.2.2 GN is doing, but that version of the OS pre-dates the CTS tests, so behavior is hard to predict. I think you can just check the flags for EOS and CODEC_CONFIG, and not try to render those -- this might cause you to drop the final frame if EOS isn't sent on an empty frame though. Not sure offhand what happens if you try to render empty EOS. – fadden Feb 05 '14 at 15:53
  • Thanks, that worked. My findings of 4.2.2 GN: size is always 0, offset is always 0, flags either 0 or EOS (but never CODEC_CONFIG), presentationTimeUs valid except when EOS (in which case it will be some random old pts that should not be trusted). Don't render when EOS since it's not a real frame. I also found that not as much texture memory could be used (even though you can mmap and use the same amount), but this seems to be a known issue with the device. Maybe I'm not surprised there are no further OS updates for the GNex. – Xargs Feb 06 '14 at 07:49
  • Galaxy Nexus is based on TI OMAP 4460. As noted in http://en.wikipedia.org/wiki/OMAP, TI essentially left the mobile space in late 2012. GN did get updated to Android 4.3 in mid-2013, but future improvements are unlikely. At any rate, I'm glad you found something that works. – fadden Feb 06 '14 at 15:38