2

I am trying to get specific frames at specific times as images from a movie using MediaExtractor and MediaCodec. I can do it successfully if:

  • I use extractor.seekTo(time, MediaExtractor.SEEK_TO_PREVIOUS_SYNC); , however, this only gives the nearest sync frame not the target frame.
  • I sequentially extract all frames using extractor.advance(); , but I need to get the target frame not all.

So, I try the following:

extractor.seekTo(time, MediaExtractor.SEEK_TO_PREVIOUS_SYNC);
while(extractor.getSampleTime()<time /*target time*/) extractor.advance();

This provides the correct frame, but for some reason the image is corrupted. It looks like the correct image (the one I get from the successful cases), but with some pixelation and a strange haze.

The while-loop is the only thing that is different between the successful cases and the corrupted ones. What to do to advance MediaExtractor to a specific time (not just sync time) without getting a corrupted image?

Mohamed_AbdAllah
  • 5,311
  • 3
  • 28
  • 47
  • Are you passing all of the frames to the decoder in your `while` loop, or just extracting the target frame and handing that to the decoder? (i.e. are you handing a P-frame to the decoder?) – fadden Jan 14 '14 at 01:11
  • @fadden Thank you for always being there for MediaCodec questions :) I loop as stated in the question until extractor is at the target time and then pass extractor.readSampleData() to feed the decoder. To be honest, I don't know what is a P-frame – Mohamed_AbdAllah Jan 14 '14 at 08:06
  • Basically, I initially use extractor.advance(); to get all the times, then pass these times to a custom adapter. The getView() of this adapter takes the time, pass it the while loop stated to set the extractor to the time I got from the adapter, then pass this time to MediaCodec to extract the frame at that time – Mohamed_AbdAllah Jan 14 '14 at 09:35
  • 2
    See http://en.wikipedia.org/wiki/Video_compression_picture_types . Each frame isn't compressed individually, so you have to start decoding from an I-frame. As far as I can tell, the terms "I frame", "key frame", and "sync frame" are used somewhat interchangeably. So I believe you have to start decoding from the nearest sync frame. – fadden Jan 14 '14 at 15:39
  • @fadden Hi, I am trying to feed the extractor output to MediaMuxer, however, it fails at muxer.stop(). I check your answer to [this](http://stackoverflow.com/questions/19505845/mediamuxer-error-failed-to-stop-the-muxer) question, however, I am sure both Video and Audio tracks have "csd-0" field (I try to use extractor to feed both video and audio to the Muxer with same formats received from extractor) – Mohamed_AbdAllah Jan 20 '14 at 10:06
  • Create a new question here with the logcat output. `MediaMuxer` spits out a bunch of stuff when `close()` is called; show that and any exception that was thrown. A short snippet showing how you're feeding the muxer would also be useful. – fadden Jan 20 '14 at 17:02
  • I submitted a new [question](http://stackoverflow.com/questions/21341169/mediamuxer-fails-to-stop-if-csd-1-not-exist) – Mohamed_AbdAllah Jan 25 '14 at 18:34
  • Before I ask a new question - should this technique (grabbing the closest previous sync frame to your desired time and running the encoder forward from there) perform well enough to do real-time scrubbing back and forth through a video? I'm seeing terrible lag, so just wanting to clarify if I'm expecting too much. :) If this should work, I'll open a new question to discuss. Thanks! – Glen Jan 27 '14 at 20:41
  • For me, A frame can take 500 - 1000 ms for an 1080p video (including seeking, decoding and conversion to Bitmap). I think I can enhance more by reducing GCs – Mohamed_AbdAllah Jan 27 '14 at 22:15

1 Answers1

2

Thanks to fadden comment, I have to keep feeding the encoder since the I-frame has the full picture and the P and B frames have differences (this is how compression is achieved). So I need to start with an I-frame (it was same as sync frame) and keep feeding the other frames to the decoder to receive the full image.

Mohamed_AbdAllah
  • 5,311
  • 3
  • 28
  • 47
  • I'm facing the [same kind of situation](http://stackoverflow.com/questions/24660469/copy-consecutive-frames-from-video). How do you get the specific time in the first place? Because I'm using a similar solution, but I never get the expected frame. I suspect the time I receive from the video view is not accurate, or my while loop is not working right. – mbmc Jul 13 '14 at 16:40
  • Could you give more details about how you "keep feeding" it from the I-frame until you reach the frame you want? – MindTrip Jul 13 '14 at 20:45
  • @Mohamed_AbdAllah: Would MediaExtractor.SEEK_TO_PREVIOUS_SYNC seek video to I frame? and how exactly do you keep feeding from I Frame? – Mahantesh M Ambi May 07 '15 at 03:38
  • Yes it does. I keep feeding using extractor.advance() to get the following P/B frame. – Mohamed_AbdAllah May 07 '15 at 23:12