5

I have an interesting problem I need to research related to very low level video streaming.

Has anyone had any experience converting a raw stream of bytes(separated into per pixel information, but not a standard format of video) into a low resolution video stream? I believe that I can map the data into RGB value per pixel bytes, as the color values that correspond to the value in the raw data will be determined by us. I'm not sure where to go from there, or what the RGB format needs to be per pixel.

I've looked at FFMPeg but it's documentation is massive and I don't know where to start.

Specific questions I have include, is it possible to create CVPixelBuffer with that pixel data? If I were to do that, what sort of format for the per pixel data would I need to convert to?

Also, should I be looking deeper into OpenGL, and if so where would the best place to look for information on this topic?

What about CGBitmapContextCreate? For example, if I went I went with something like this, what would a typical pixel byte need to look like? Would this be fast enough to keep the frame rate above 20fps?

EDIT:

I think with the excellent help of you two, and some more research on my own, I've put together a plan for how to construct the raw RGBA data, then construct a CGImage from that data, in turn create a CVPixelBuffer from that CGImage from here CVPixelBuffer from CGImage.

However, to then play that live as the data comes in, I'm not sure what kind of FPS I would be looking at. Do I paint them to a CALayer, or is there some similar class to AVAssetWriter that I could use to play it as I append CVPixelBuffers. The experience that I have is using AVAssetWriter to export constructed CoreAnimation hierarchies to video, so the videos are always constructed before they begin playing, and not displayed as live video.

Community
  • 1
  • 1
Matt Foley
  • 921
  • 1
  • 8
  • 24
  • 3
    Have a deeeeeeeeep look into AVFoundation, especially into AVAssetWriter. After reading that documentation, a good starting point might be "[How do I export an UIImage array as a movie](http://stackoverflow.com/questions/3741323/how-do-i-export-uiimage-array-as-a-movie/3742212)". – Till Mar 30 '13 at 19:34
  • I've done quite a bit of work with AVAssetWriter, but my current problem is that I don't know how to structure the raw pixel data. – Matt Foley Mar 31 '13 at 03:05
  • 1
    That is the issue indeed. You have to figure out how to construct a Pixel Buffer with the raw that that you are receiving from the stream. That will depend on the format, encoding and so on of the incoming data. Once you have that, check the poste that @Till recommended and AVAssetWritter. This might help CVPixelBufferRef buffer = (CVPixelBufferRef)[self pixelBufferFromCGImage:theImage size:CGSizeMake(image.size.width, image.size.height)]; if (buffer) { if(![adaptor appendPixelBuffer:buffer withPresentationTime:CMTimeMake(frame, fps)]){ – Javier Quevedo Mar 31 '13 at 13:05
  • I added more information to a comment, and then had to add it to the answer due to length. Thank you both, I'm getting closer but I still have a few questions. – Matt Foley Mar 31 '13 at 18:59

2 Answers2

5

I've done this before, and I know that you found my GPUImage project a little while ago. As I replied on the issues there, the GPUImageRawDataInput is what you want for this, because it does a fast upload of RGBA, BGRA, or RGB data directly into an OpenGL ES texture. From there, the frame data can be filtered, displayed to the screen, or recorded into a movie file.

Your proposed path of going through a CGImage to a CVPixelBuffer is not going to yield very good performance, based on my personal experience. There's too much overhead when passing through Core Graphics for realtime video. You want to go directly to OpenGL ES for the fastest display speed here.

I might even be able to improve my code to make it faster than it is right now. I currently use glTexImage2D() to update texture data from local bytes, but it would probably be even faster to use the texture caches introduced in iOS 5.0 to speed up refreshing data within a texture that maintains its size. There's some overhead in setting up the caches that makes them a little slower for one-off uploads, but rapidly updating data should be faster with them.

Brad Larson
  • 170,088
  • 45
  • 397
  • 571
  • Can't even tell you how much I appreciate your help. I replied over there, but to keep it on Stack in case anyone else comes looking I just wanted to make sure I was understanding the class correctly. I'm guessing that I call uploadBytes per frame on GPUImageRawDataInput? From there I add the GPUImageFilter I use as a target of the RawDataInput and then add the GPUImageView as a target of the filter I use? If I wanted to skip the filter, I could just add the RawDataInput as a target of the GPUImageView, or would I use the RawDataOutput somehow? – Matt Foley Apr 06 '13 at 19:28
  • 1
    @MattFoley - Yes, you'll need to use `-uploadBytes:` with each new frame that comes in. You'll also need to follow that with `-processData` to get it to propagate down the rest of the pipeline once uploaded. You can go directly from the raw data input to a GPUImageView, if display is all you care about. – Brad Larson Apr 08 '13 at 02:56
  • Hi Brad, a year later and I'm still working on this software in my free time. You mention that if display is all that mattered that I simply needed to call "processData:". I'm happy to say that we've had playback of the raw data working for quite some time, but currently I'm working on integrating GPUImageMovieWriter with the GPUImageRawDataInput you helped me with and running into a "12902" status on the AVAssetWriter when starting recording. This is my code:http://pastebin.com/LD6ZcJtN Are there extra steps needed I am missing? Thanks for everything Brad! – Matt Foley Jul 10 '14 at 17:52
  • Adding to this comment, I found that there is a deeper issue with the size of video I'm recording. I was receiving the following outputs to the device's console complaining that 80x80 did not meet a minimum dimensions requirement for the h264 codec that was not piped to Xcode: http://pastebin.com/bVQ7PJ12 – Matt Foley Jul 10 '14 at 19:18
1

My 2 cents:

I made an opengl game which lets the user record a 3d scene. Playback was done via replaying the scene (instead of playing a video because realtime encoding did not yield a comfortable FPS.

There is a technique which could help out, unfortunately I didn't have time to implement it: http://allmybrain.com/2011/12/08/rendering-to-a-texture-with-ios-5-texture-cache-api/

This technique should cut down time on getting pixels back from openGL. You might get an acceptable video encoding rate.

cantgetright82
  • 139
  • 4