CVOpenGLESTextureCacheCreateTextureFromImage instead of glReadPixels

Question

For 3 full days, I have been trying to improve the performance of my AVAssetWriter which is based on glReadPixels. I have gone through Apple's RosyWriter and Camera Ripple code and Brad Larson's GPUImage but I am still scratching my head. I've also been trying to use the implementations put down in these links:

rendering-to-a-texture-with-ios-5-texture-cache-api

faster-alternative-to-glreadpixels-in-iphone-opengl-es-2-0

...and many more but no matter what I try, I just can't get it to work. Either the video end up not being processed, or it comes out black or I get various errors. I won't go through all of it here.

To simplify my question, I thought I'd focus around just grabbing a snapshot from my onscreen openGL preview FBO. If I can just get one single implementation of this working, I should be able to work out the rest. I tried the implementation from the first link above which looks something like this:

CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [glView context], 
                           NULL, &texCacheRef);

CFDictionaryRef empty = CFDictionaryCreate(kCFAllocatorDefault,
                           NULL,
                           NULL,
                           0,
                           &kCFTypeDictionaryKeyCallBacks,
                           &kCFTypeDictionaryValueCallBacks);

CFMutableDictionaryRef attrs = CFDictionaryCreateMutable(kCFAllocatorDefault,
                                  1,
                                  &kCFTypeDictionaryKeyCallBacks,
                                  &kCFTypeDictionaryValueCallBacks);

CFDictionarySetValue(attrs,
                     kCVPixelBufferIOSurfacePropertiesKey,
                     empty);

CVPixelBufferRef renderTarget = NULL;
CVPixelBufferCreate(kCFAllocatorDefault,
                    width,
                    height,
                    kCVPixelFormatType_32BGRA,
                    attrs,
                    &renderTarget);

CVOpenGLESTextureRef renderTexture;
CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault,
                                              texCacheRef,
                                              renderTarget,
                                              NULL,
                                              GL_TEXTURE_2D,
                                              GL_RGBA,
                                              width,
                                              height,
                                              GL_BGRA,
                                              GL_UNSIGNED_BYTE,
                                              0,
                                              &renderTexture);

CFRelease(attrs);
CFRelease(empty);
glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

GLuint renderFrameBuffer;
glGenRenderbuffers(1, &renderFrameBuffer);
glBindFramebuffer(GL_FRAMEBUFFER, renderFrameBuffer);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
                       GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);

//Is this really how I pull pixels off my context?
CVPixelBufferLockBaseAddress(renderTarget, 0);
buffer = (GLubyte *)CVPixelBufferGetBaseAddress(renderTarget);
CVPixelBufferUnlockBaseAddress(renderTarget, 0);

What exactly is supposed to happen here? My buffer ends up being a bunch of zero's so I guess I need to do something additional to pull the pixels from the context? ...or what am I missing?

All I want to achieve is a faster equivalent of what I am using today:

int pixelsCount = w * h;
buffer = (GLubyte *) malloc(pixelsCount * 4);
glReadPixels(0, 0, w, h, GL_RGBA, GL_UNSIGNED_BYTE, buffer);

You don't show any actual rendering in the above code. Once you have your texture-backed FBO, and the texture for that has been provided by a texture cache, you need to render your content into that FBO. Only then can you read back the pixel data from the texture used to back your FBO. You may need to insert a `glFinish()` right before you read back the pixel data from your buffer, in order to guarantee that OpenGL ES has had the time to finish rendering into your texture. — Brad Larson, Oct 08 '12 at 01:26
Ah, can't believe I missed that. I was under the impression rendering to the main FBO would be enough and then I could pull from it similarly to glReadPixels. I just tried replacing my FBO with this one and that did the trick. Thanks a lot! If you put your comment as a response, I will accept it. — BlueVoodoo, Oct 08 '12 at 10:30
Although I got it working, I am still a bit confused about the CVOpenGLESTextureCacheCreate call. The EAGLContext that is being passed in here, is that expected to be from another FBO which I have rendered to already ...or could a new context be used in order to only render once (directly to the one in the above code)? As soon as I use another context, I only get back zero's in my buffer. — BlueVoodoo, Oct 08 '12 at 12:02
For the record, just benchmarked this method on my asset writer (h264/aac). Writes out the asset at roughly 3/4 of the time compared to the glReadPixel method. — BlueVoodoo, Oct 08 '12 at 15:29

score 1 · Accepted Answer · answered Oct 10 '12 at 10:43

1

As Brad pointed out, I misunderstood the concept and wasn't doing any actual rendering. Worked fine when I added it.

answered Oct 10 '12 at 10:43

BlueVoodoo

3,626
5
29
37

I suffer from the same problem as you meet. and my screen is black... can you tell me how do you solve it? – CPT Jul 13 '14 at 14:49
My issue was that I wasn't rendering before I pulled the buffer. In other words, I did not call glDrawArrays. – BlueVoodoo Jul 14 '14 at 17:27
thanks for reply! I try put such `glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);` code in my project, it still can not work. Also I refer the GPUImage project on github, i think it may be some difficult for me. may you give some hints about rendering details? source code is better...thanks. – CPT Jul 15 '14 at 05:47
It's difficult to explain without having access to your code. Did you get it working with glReadPixels but not with the above technique? If so, you basically need to do all the same steps as the glReadPixel approach. This is not a replacement for the rendering. If not, you should to aim to get this working with glReadPixels first. That is a far easier approach. – BlueVoodoo Jul 15 '14 at 15:24
yes, it is successful when using the glReadPixel. someone says using the texture cache will be more faster, but it doesnot work for me. I post my code in [link](https://github.com/Dominic--/Stackoverflow/blob/master/powersmart_interface.m). Can you have a look if possible, in my code, `initdata` will be called when the game start, and `powersmart_callback` will be called in every update frame. in this function the screen will be saved as a picture. Now with using this method, the picture will saved successfully, but the screen is still black...could you help me.? – CPT Jul 17 '14 at 04:31

CVOpenGLESTextureCacheCreateTextureFromImage instead of glReadPixels

1 Answers1