Play Video and audio using MediaCodec and MediaExtractor

Question

I am trying to play a video using MediaCodec. The video is playing but the audio is mute.I want to unmute the audio.After research I found that we need to decode the audio and video separately.I want to use only one Mediacodec and MediaExtractor object to play video and audio.How is this achievable? I used the following code for playing video.

        extractor = new MediaExtractor();
        extractor.setDataSource(SAMPLE);

        for (int i = 0; i < extractor.getTrackCount(); i++) {
            MediaFormat format = extractor.getTrackFormat(i);
            String mime = format.getString(MediaFormat.KEY_MIME);
            if (mime.startsWith("video/")) {
                extractor.selectTrack(i);
                decoder = MediaCodec.createDecoderByType(mime);
                decoder.configure(format, surface, null, 0);
                break;
            }
        }

        if (decoder == null) {
            Log.e("DecodeActivity", "Can't find video info!");
            return;
        }

        decoder.start();

        ByteBuffer[] inputBuffers = decoder.getInputBuffers();
        ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
        BufferInfo info = new BufferInfo();
        boolean isEOS = false;
        long startMs = System.currentTimeMillis();

        while (!Thread.interrupted()) {
            if (!isEOS) {
                int inIndex = decoder.dequeueInputBuffer(10000);
                if (inIndex >= 0) {
                    ByteBuffer buffer = inputBuffers[inIndex];
                    int sampleSize = extractor.readSampleData(buffer, 0);
                    if (sampleSize < 0) {
                        // We shouldn't stop the playback at this point, just pass the EOS
                        // flag to decoder, we will get it again from the
                        // dequeueOutputBuffer
                        Log.d("DecodeActivity", "InputBuffer BUFFER_FLAG_END_OF_STREAM");
                        decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                        isEOS = true;
                    } else {
                        decoder.queueInputBuffer(inIndex, 0, sampleSize, extractor.getSampleTime(), 0);
                        extractor.advance();
                    }
                }
            }

            int outIndex = decoder.dequeueOutputBuffer(info, 10000);
            switch (outIndex) {
            case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
                Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");
                outputBuffers = decoder.getOutputBuffers();
                break;
            case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
                Log.d("DecodeActivity", "New format " + decoder.getOutputFormat());
                break;
            case MediaCodec.INFO_TRY_AGAIN_LATER:
                Log.d("DecodeActivity", "dequeueOutputBuffer timed out!");
                break;
            default:
                ByteBuffer buffer = outputBuffers[outIndex];
                Log.v("DecodeActivity", "We can't use this buffer but render it due to the API limit, " + buffer);

                // We use a very simple clock to keep the video FPS, or the video
                // playback will be too fast
                while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
                    try {
                        sleep(10);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                        break;
                    }
                }
                decoder.releaseOutputBuffer(outIndex, true);
                break;
            }

            // All decoded frames have been rendered, we can stop playing now
            if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                Log.d("DecodeActivity", "OutputBuffer BUFFER_FLAG_END_OF_STREAM");
                break;
            }
        }

        decoder.stop();
        decoder.release();
        extractor.release();
    }

score 4 · Answer 1 · edited Mar 22 '19 at 13:15

4

You can use one extractor but two MediaCodec instances for audio and video playback. Only one MediaCodec instance can't do that. For demuxer one extractor is ok, it demuxes file into an audio stream and a video stream. But audio and video streams should use different decoders, like audio AAC, video AVC etc... And MediaCodec can't change decode type dynamically, so you should create different MediaCodec instances for each stream. A player should consider AV synchronization... You can refer: https://github.com/saki4510t/AudioVideoPlayerSample

edited Mar 22 '19 at 13:15

Anton

560
6
21

answered Feb 27 '17 at 10:30

Hanson.tian

41
3

Thank you for your feedback.I used two different mediacodec instance and it works fine. – Mahesh Ambekar May 03 '17 at 13:07
@MaheshAmbekar Can you please elaborate? I have the same issue – Nandha Kumar Oct 01 '18 at 04:46
1

Use two MediaExtractor, one for audio and other one for video and keep two separate threads for executing the same. – Mahesh Ambekar Nov 15 '18 at 11:13
In https://github.com/saki4510t/AudioVideoPlayerSample/blob/master/app/src/main/java/com/serenegiant/media/MediaMoviePlayer.java from the reference in the answer there are two media extractor used: `mVideoMediaExtractor` and `mAudioMediaExtractor` – sergtk May 21 '19 at 17:24

Play Video and audio using MediaCodec and MediaExtractor

1 Answers1