1

I am currently attempting to parse H264 data from an RTP stream and then send it to the MediaCodec to render on a SurfaceView for Android.

However, I'm not certain how to:

  • build the H264 slices properly from the RTP packets
  • send the H264 slices to the media codec once they are assembled into slices

I have not seen any examples of this implemented in a clear and concise way and I haven't found the MediaCodec docs to be at all helpful.

Anyone have any experience in this domain?

void videoCodec(ByteBuffer input, int flags) {

    bufferInfo.set(0, 0, 0, flags);

    int inputBufferId = codec.dequeueInputBuffer(10000);

    if (inputBufferId >= 0) {

        //put data
        ByteBuffer inputData = inputBuffers[inputBufferId];

        inputData.clear();
        inputData.put(input);

        //queue it up
        codec.queueInputBuffer(inputBufferId, 0, input.limit(), 0, flags);
    }

    int outputBufferId = codec.dequeueOutputBuffer(bufferInfo, 10000);

    if (outputBufferId >= 0) {
        // outputBuffers[outputBufferId] is ready to be processed or rendered.
        Timber.e("Rendering Data with Index of: %s", outputBufferId);
        codec.releaseOutputBuffer(outputBufferId, true);

    } else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
        outputBuffers = codec.getOutputBuffers();
    } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
        // Subsequent data will conform to new format.
        //format = codec.getOutputFormat();
    }
}

 MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 1920, 1080);
                    codec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
                    codec.configure(format, surfaceVideo.getHolder().getSurface(), null, 0);
                    codec.start();


                    inputBuffers = codec.getInputBuffers();
                    outputBuffers = codec.getOutputBuffers();

      while (streaming) {

          //receive RTP Packet
          h264Parser(rtpPacket.getPayload());

      }

And the the h264Parser looks something like this: 

void h264Parser(byte[] payload) {

    int packetType = (byte) payload[0] & (byte) 0x1F;
    boolean startBit = (payload[1] & 0x80) != 0;
    boolean endBit = (payload[1] & 0x40) != 0;
    int flags = 0;

    switch (packetType) {
        case 7:
            pps = new ByteArrayOutputStream();
            pps.write(prefix);
            pps.write(payload);
            break;
        case 8:
            if (pps.size() > 0) {
               pps.write(payload);
               hasPps = true;
               flags = MediaCodec.BUFFER_FLAG_CODEC_CONFIG;
               payload = pps.toByteArray();
               //Send packet to decoder
               videoCodec(ByteBuffer.wrap(payload), flags);
            break;
        case 28:

            if (hasPps) {
                if (startBit) {
                    baos = new ByteArrayOutputStream();
                    baos.write(prefix);
                    baos.write(payload);
                } else if (endBit) {
                        if(baos != null) {
                            baos.write(payload);
                            flags = MediaCodec.BUFFER_FLAG_KEY_FRAME;
                            payload = baos.toByteArray();
                            //Send packet to decoder
                            videoCodec(ByteBuffer.wrap(payload), flags);
                            hasPps = false;
                } else {
                        if(baos != null ) {
                            baos.write(payload);
                        }
                }
            }

            break;
        case 1:
            break;
        default:
    }
androidDeving
  • 75
  • 1
  • 7
  • you can use some library to play live-videos. I have worked with Vitamio and its really easy to use : https://www.vitamio.org/en/ – WannaBeGeek Apr 26 '16 at 17:10
  • 1
    I'm not the author of this library but while doing some similar research I found this: [AndroidStreamingClient](https://github.com/ekumenlabs/AndroidStreamingClient) Specifically see the code here: https://github.com/ekumenlabs/AndroidStreamingClient/blob/master/android_streaming_client/src/main/java/com/c77/androidstreamingclient/lib/rtp/RtpMediaExtractor.java Which seems to do what you are looking for. – Matt Wolfe Jun 02 '17 at 18:02

1 Answers1

1

As far as i remember MediaCodec uses full acess units, not only slices (someone correct me of i'm wrong)

So you have to build a complete acess unit form RTP and feed it to the decoder (sadly i have no experience with RTP and cannot help you with building one).

You send the access units to the decoder as follows:

Dequeue a inputbuffer

int inputBufferIndex = decoder.dequeueInputBuffer(TIMEOUT_USEC); ByteBuffer inputBuffer = videoDecoderInputBuffers[videoInputBufIndex];

Fill it with your access unit

inputBuffer.put(acessUnit); inputBuffer.flip();

Queue the buffer for decoding

decoder.queueInputBuffer(inputBufferIndex,0,inputBuffer.limit(), 0, FLAGS);

I hope this helps a little

chris
  • 106
  • 3
  • Thanks, Chris! That is helpful. I'm digging in now. – androidDeving Apr 27 '16 at 14:23
  • I keep getting an outputBufferId of -1. I suspect that means there are no available output buffers to dequeue. Any thoughts? – androidDeving Apr 27 '16 at 18:28
  • Exactly, this is also the case if you feed the mediacodec input that cant be processed. Keep in mind that you have to feed the SPS and PPS to the decoder. You could do that via the MediaFormat you pass to the decoder (`format.setByteBuffer("csd-0", ByteBuffer.wrap(header_sps)); format.setByteBuffer("csd-1", ByteBuffer.wrap(header_pps));`) or by sending them as first buffer using the flag `BUFFER_FLAG_CODEC_CONFIG` – chris Apr 29 '16 at 12:19
  • I'm curious about the `inputBuffer.flip()`. Does the decoder need a certain endian? Also, I've seen implementations that clear the buffer before putting more data in it. – androidDeving May 03 '16 at 13:38
  • You get an already empty Buffer (might be device specific, you could check position and limit for verification) but clearing never hurts since it only sets position to 0 and limit to capacity. The `inputBuffer.flip()` only prepares the buffer to be read as it sets limit to position and position to 0 – chris May 04 '16 at 14:16
  • None of this is working. Is there anywhere that I might find a full example of how this works? – androidDeving May 04 '16 at 19:00
  • Good tutorial sites for pre API 21 implementations are [link](http://bigflake.com/mediacodec/) and [link](https://github.com/google/grafika) . You could be a bit more specific about your problems, what is the output you get, how does your SPS and PPS look like, do your Access Units begin with 0x00 0x00 0x00 0x01 – chris May 06 '16 at 08:32
  • I don't mean to sounds daft, but how does one prefix a byte array with 0x00 0x00 0x00 0x01? is that just 0000000 00000000 0000000 00000001? Also, does the SPS PPS units need to be sent separately or together as one unit with the hex prefix? – androidDeving May 12 '16 at 18:10
  • No need to feel daft, because working with hex in java is a pain. I usually use a function wich converts a string to a bytearray [link](http://stackoverflow.com/a/140861/6205344) – chris May 13 '16 at 10:25
  • I think u can do both, (check the documentation on [link](http://developer.android.com/reference/android/media/MediaCodec.html) under Initialization) Also make sure to add the complete NAL Unit, it looks something like this for SPS: {0x00, 0x00, 0x00, 0x01, 0x67, ...} and for PPS something like this: {0x00, 0x00, 0x00, 0x01, 0x68, ...} – chris May 13 '16 at 10:47
  • Thank you, Chris. I'll give this a whirl today. – androidDeving May 13 '16 at 13:33
  • I'm sending everything properly, I believe, however I'm getting an OutputBufferId of -1 and a log message "not in avi mode". – androidDeving May 13 '16 at 14:21
  • How many frames have you sent to the decoder? Most decoders will wait until you have sent a few frames. Can you post the loop and the codec initialization? I'm also wondering about this: `bufferInfo.set(0, 0, 0, flags);` what are you trying to accomplish with this ()? Also have a look at this [link](http://stackoverflow.com/questions/32841513/mediacodec-crash-on-high-quality-stream/32845019#32845019) – chris May 13 '16 at 17:04
  • I'm receiving H264 data over an RTP stream. That stream is itself in a while loop: `while(streaming) { //receive rtp data}`. For each packet I receive, I strip the header. Next I determine the packet type to find the PPS and SPS. I combine those two and prefix them with 0x00 0x00 0x00 0x01. If it a type 28 unit I test if it is a start packet, end packet. If it's neither it gets concatenated after the start packet and until the end packet. Once the entire access unit is assembled I prefix it with 0x00 0x00 0x00 0x01. Then I send it off to the decoder. – androidDeving May 13 '16 at 17:30
  • I can't find an obvious mistake here but some things that seem a bit problematic to me. First, are you sure that every frame is a Key Frame? Second, usually SPS and PPS are only passed at the beginning, so if they don't change, don't update them. You could post some entire buffers you send to processing (config and AU's) and check if everything is valid (my guess is that there is some mistake in the repackaging). Lastly some design suggestions, you should always check for output independently, not only if there is an AU completed. Also i suggest building AU's separately and add them to a queue – chris May 18 '16 at 13:22
  • @androidDeving, if this post's advice was useful then please upvote it to show "thanks" (up arrow near tick mark)... Also 0x00 0x00 0x00 0x001 is just another way of saying 4 bytes of value `00 00 00 01` (note: a single integer is spread over 4 bytes). As @Chris said, just send the SPS & PPS to decoder, from there onwards just remove RTP header & join bytes of slices (usually 2) to make a single AU (video frame) but it can be 3 slices & also usually the number doesn't change during video duration). When you have a single AU send to decoder for image output. If problems, show bytes of 1 AU.. – VC.One May 31 '16 at 00:29