7

Need to decode h264 stream and get the pixel buffers

I know its possible with video tool box on iOS 8

1.How do I convert the h264 stream to CMSampleBufferRef ?

2.How do I use the video tool box to decode?

Rajesh
  • 10,318
  • 16
  • 44
  • 64
sivan
  • 91
  • 1
  • 6
  • Did you read the documentation? Did you watch the WWDC video on the subject? Why have you tried? – szatmary Sep 24 '14 at 16:28
  • I am knew to the video & video streaming world. Yes I saw the video from WWDC 2014, but right now its not enough cause I miss some details, which may be obvious to you :) – sivan Sep 25 '14 at 19:31
  • 1
    Oh, it very obvious to me you are missing details. In fact you don't have enough details in your question to even answer it. What format is your h264 stream in? Is it annex B? Do you have the pointers to the SPS/PPS? – szatmary Sep 25 '14 at 20:00
  • I get the h264 by udp from external device to the iphone, need to send the pixel buffer to sdk I am using for video conference – sivan Sep 26 '14 at 04:49
  • I have avcc stream coming from udp socket – sivan Sep 28 '14 at 09:35
  • Where can I find the aforementioned WWDC video on VideoToolbox? – Philip Bulley Mar 10 '15 at 20:16
  • here https://developer.apple.com/videos/wwdc/2014/#513 look for the direct access to video encoding and decoding session – sivan Mar 11 '15 at 21:21

2 Answers2

7

I assume you get the stream in Annex B format, if it is already in AVCC format (read MP4), then you can you the AssetReader and do not need to do much.

For an Annex B stream (this is what ppl. often call raw h264 stream).

  1. extract SPS/PPS NAL units and create a parameter set from then. You receive them periodically. They contain information for the decode how a frame is supposed to be decoded.

  2. create the TimingInfo Array with the duration (you can take it from parsing the VUI part of SPS) and presentation time stamp and decoding timestamp.Iif the stream is received as MPEG2 TS take the timestamps from the PESr. If not just supply missing info based on your calculations.

  3. Wrap the VLC NAL units in a CMBlockBuffer. You can put more then one into them. If you receive your stream over RTP that might fragment the NAL units make sure every NAL unit is complete.

  4. When wrapping the NAL unit in a CMBlockbuffer replace the 3- or 4-byte start code with a length header.

  5. Supply the information to CMSampleBufferCreate and you can decode the frame in VTDecompressionSession

There is a presetation from WWDC available that explains these steps a bit more in detail ans also provide sample code.

scythe42
  • 483
  • 2
  • 6
  • From the documentation AssetReader can't help with network stream. Hope I am wrong. AVAssetRedaer is initialised with AVAsset. Didn't find a way to make it work with a stream. Saw some posts about writing the stream to a file and reading the file with AVAssetReader, but that is not working well, and I need good performance and low battery consumption so... – sivan Oct 01 '14 at 12:55
  • Correct, AssetReader cannot deal with Annex B streams. You have to convert to AVCC format (basically MP4 container). This is basically the same as above but instead of decoding you write it to a file. But this make only sense if you want to keep the stream you received and want to play it multiple times later with IOS/OSX the easy way. – scythe42 Oct 01 '14 at 20:09
  • I have succeeded in decoding the h264 stream, now i wanna display the stream to the iPhone's screen. do you know how ? – sivan Oct 14 '14 at 11:29
  • I use `CADisplayLink` for my presenter to handle the queue of decoded pictures of a `VTDecompressionSession`. On every vsync I render a new frame with OpenGLES to a layer. I do this, so I can do color corrections, handle OSDs etc. If such an approach makes sense for you depends on your code and what you want to try to achieve in the end. Video Toolbox provides a lof stuff here. If you share some of your code here and explain what you want to achieve next it it would be easier to point you at some classes/methods that can do the job for you displaying what's now inside CVPixelBuffers – scythe42 Oct 14 '14 at 13:59
  • 1
    trying to convert the CVImageBufferRef to CGImageRef and display it in a UIImageView. I get error in CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst); the error is - CGBitmapContextCreate: invalid data bytes/row: should be at least 1280 for 8 integer bits/component, 3 components, kCGImageAlphaPremultipliedFirst – sivan Oct 14 '14 at 14:42
  • @sivan I also had trouble converting from CVImageBufferRef to UIImage (or other formats). I know this is a late reply, but try the method that I posted [here](http://stackoverflow.com/a/29170802/3841734) if you're still having trouble. – Olivia Stork Mar 27 '15 at 16:04
3

Try this working code.Supply the encoded CMSampleBufferRef to sampleBuffer.

if(!decompressionSession)
    {
    CMFormatDescriptionRef formatDescription=CMSampleBufferGetFormatDescription(sampleBuffer);
    decompressionSession = NULL;
    VTDecompressionOutputCallbackRecord callBackRecord;
    callBackRecord.decompressionOutputCallback=didDecompress;
    callBackRecord.decompressionOutputRefCon = (__bridge void *)self;
    OSStatus status1= VTDecompressionSessionCreate(kCFAllocatorDefault, formatDescription, NULL, NULL, &callBackRecord, &decompressionSession);
    }
    else
    {
        VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression;
        VTDecodeInfoFlags flagOut;
        VTDecompressionSessionDecodeFrame(decompressionSession, sampleBuffer, flags, NULL, &flagOut);
        VTDecompressionSessionWaitForAsynchronousFrames(decompressionSession);

    }



Decompression all back

static void didDecompress( void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration ){
 if(status==noErr)
    {
        NSLog(@"SUCCESS PROCEED FROM HERE !!!!");
    }
}

// Keep in mind , you provide the correct presentation time while encoding.Here i am providing you the encoding details..

//ENCODING-------------------ENCODING---------------ENCODING
if(!_compression_session)
{
  NSDictionary* pixelBufferOptions = @{
                                          (NSString*) kCVPixelBufferWidthKey : @(widthOFCaptureImage),
                                          (NSString*) kCVPixelBufferHeightKey : @(heightOFCaptureImage),
                                          (NSString*) kCVPixelBufferOpenGLESCompatibilityKey : @YES,
                                          (NSString*) kCVPixelBufferIOSurfacePropertiesKey : @{}};
_compression_session=NULL;
 CFMutableDictionaryRef encoderSpecifications = NULL;
  err = VTCompressionSessionCreate(
                                     kCFAllocatorDefault,
                                     widthOFCaptureImage,
                                     heightOFCaptureImage,
                                     kCMVideoCodecType_H264,
                                     encoderSpecifications,
                                     (__bridge CFDictionaryRef)pixelBufferOptions,
                                     NULL,
                                     compressionCallback,
                                     (__bridge void *)self,
                                     &_compression_session);
}
else
{


 CMTime presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(sampleBufferIs);
  CVPixelBufferRef pixelbufferPassing= CMSampleBufferGetImageBuffer(sampleBufferIs);
          OSStatus status1= VTCompressionSessionEncodeFrame(_compression_session, pixelbufferPassing, presentationTimeStamp, kCMTimeInvalid, NULL, NULL, NULL);
 VTCompressionSessionEndPass(_compression_session, NO, NULL);
}

//ENCODING CALL BACK-----------------------------------------

  static void compressionCallback(void *outputCallbackRefCon,
                             void *sourceFrameRefCon,
                             OSStatus status,
                             VTEncodeInfoFlags infoFlags,
                             CMSampleBufferRef sampleBuffer ){
    }

//Best wishes :) happy coding :)

good4pc
  • 711
  • 4
  • 17