CIImage created from CVPixelBufferRef draws rotated and shrunk in GLKView

Question

I'm trying to grab video with an AVCaptureSession, process the video in the callback (eventually), then render the results into my GLKView. The code below works but the image in my GLKView is rotated 90 degrees and shrunk by 50%.

The glContext is created with [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];

My coreImageContext is created with [CIContext contextWithEAGLContext:glContext];

- (void)captureOutput:(AVCaptureOutput *)captureOutput 
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer 
       fromConnection:(AVCaptureConnection *)connection
{
    // process the image
    CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
    CIImage *image = [CIImage imageWithCVPixelBuffer:pixelBuffer];

    // display it (using main thread)
    dispatch_async( dispatch_get_main_queue(), ^{
        // running synchronously on the main thread now
        [self.coreImageContext drawImage:image inRect:self.view.bounds fromRect:[image extent]];
        [self.glContext presentRenderbuffer:GL_RENDERBUFFER];
    });
}

Inserting code to perform and affine transform seems inefficient. Am I missing a setup call or parameter to prevent the rotation and scaling?

Same problem here. I used almost the same code as in your case. I ended up giving up and used affine transform to correct the rotation and to resize the image to fill the entire screen. — Di Wu, Dec 23 '11 at 09:11

score 4 · Answer 1 · edited May 23 '17 at 12:00

The video preview view for AVFoundation does rotations on the images using the graphics hardware. The native capture orientation for the back facing camera is landscape left. It is landscape right for the front facing camera. When you record a video AVFoundation will place a transform in the header of the MOV/MP4 to indicate to a player the correct orientation of the video. If you just pull the images out of the MOV/MP4 they will be in their native capture orientation. Take a look at the two example programs linked in the top post for this SO Post

score 0 · Answer 2 · answered Oct 21 '16 at 14:52

drawImage:inRect:fromRect: will scale the image to fit the destination rectangle. You just need to appropriately scale self.view.bounds:

CGFloat screenScale = [[UIScreen mainScreen] scale];
[self.coreImageContext drawImage:image inRect:CGRectApplyAffineTransform(self.view.bounds, CGAffineTransformMakeScale(screenScale, screenScale)) fromRect:[image extent]];

CIImage created from CVPixelBufferRef draws rotated and shrunk in GLKView

2 Answers2