Generate movie with UIImages using AVFoundation

Question

Many before me have shared their knowledge in stack overflow about this topic. I was able to take over much of the tips and code snippets thanks to the contribution. It all worked quite good except that it was often hard on the working memory. This time-lapse application that I am working on, was able to generate a movie out of 2000 hd images and more, but since iOS 7.1 it is having trouble generating a video out of more than 240 hd images. 240 images seems to be the limit on an iPhone 5s. I was wondering whether anybody has had these problems too and whether anybody has found solutions to it. Now to the source code.

This part iterates through saved uiimages in the apps document's directory.

if ([adaptor.assetWriterInput isReadyForMoreMediaData])
{
            CMTime frameTime = CMTimeMake(1, fps);
            CMTime lastTime=CMTimeMake(i, fps);
            CMTime presentTime=CMTimeAdd(lastTime, frameTime);

            NSString *imageFilePath = [NSString stringWithFormat:@"%@/%@",folderPathName, imageFileName];
            image = [UIImage imageWithContentsOfFile:imageFilePath] ;
            cgimage = [image CGImage];

            buffer = (CVPixelBufferRef)[self pixelBufferFromCGImage: cgimage];
            bool result = [adaptor appendPixelBuffer:buffer withPresentationTime:presentTime];


            if (result == NO)
            {
                NSLog(@"failed to append buffer %i", i);    
                _videoStatus = 0;
                success = NO;

                return success;
            }

     //buffer has to be released here or memory pressure will occur 
            if(buffer != NULL)
            {
                CVBufferRelease(buffer);
                buffer = NULL;
            }
}

This is the local method which appears to make most trouble. It gets the pixel buffer reference from cgimage.

- (CVPixelBufferRef) pixelBufferFromCGImage: (CGImageRef) image 
{

NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                         [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                         [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                         nil];



CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, CGImageGetWidth(image),
                                      CGImageGetHeight(image), kCVPixelFormatType_32ARGB, (CFDictionaryRef) CFBridgingRetain(options),
                                      &pxbuffer);

NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);    
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
 NSParameterAssert(pxdata != NULL);

CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, CGImageGetWidth(image),
                                             CGImageGetHeight(image), 8, 4*CGImageGetWidth(image), rgbColorSpace,                                               (CGBitmapInfo)kCGImageAlphaNoneSkipFirst);

NSParameterAssert(context);
CGContextConcatCTM(context, CGAffineTransformMakeRotation(M_PI));

float width = CGImageGetWidth(image);
float height = CGImageGetHeight(image);

CGContextDrawImage(context, CGRectMake(0, 0, width,
                                           height), image);

CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);


CVPixelBufferUnlockBaseAddress(pxbuffer, 0);


return pxbuffer;
}

I have been spending a lot of time on this and not moving forward. Help is much appreciated. If any more details are necessary, I am glad to provide.

Tim Bull · Answer 1 · 2014-05-15T20:47:55.200

2

I use very similar code although for slightly different reasons (I'm using AVAssetReader, grabbing frames as images and manipulating them). Net result however should be similar - I'm iterating through 1000's of images without issue.

The two things I notice that I'm doing that are different:

When you release the Buffer, you're using CVBufferRelease, I'm using CVPixelBufferRelease.
You are not releasing the CGImage using CGImageRelease.

Try rewriting this:

 //buffer has to be released here or memory pressure will occur 
        if(buffer != NULL)
        {
            CVBufferRelease(buffer);
            buffer = NULL;
        }

as:

 //buffer has to be released here or memory pressure will occur 
        if(buffer != NULL)
        {
            CVPixelBufferRelease(buffer);
            buffer = NULL;
        }
        CGImageRelease(cgImage);

Let me know how that goes.

EDIT: Here is a sample of my code, getting and releasing a CGImageRef. The Image is created from a CIImage extracted from the reader buffer and filtered.

                CGImageRef finalImage = [context createCGImage:outputImage fromRect:[outputImage extent]];

            // 2. Grab the size
            CGSize size = CGSizeMake(CGImageGetWidth(finalImage), CGImageGetHeight(finalImage));

            // 3. Convert the CGImage to a PixelBuffer
            CVPixelBufferRef pxBuffer = NULL;
            pxBuffer = [self pixelBufferFromCGImage: finalImage andSize: size];

            // 4. Write things back out.
            // Calculate the frame time
            CMTime frameTime = CMTimeMake(1, 30);
            CMTime presentTime=CMTimeAdd(currentTime, frameTime);

            [_ugcAdaptor appendPixelBuffer:pxBuffer withPresentationTime:presentTime];

            CGImageRelease(finalImage);

            CVPixelBufferRelease(pxBuffer);

edited May 15 '14 at 20:47

answered May 14 '14 at 04:26

Tim Bull

2,375
21
25

Definitely is a mistake on my behalf. CVPixelBufferRelease(buffer) makes more sense. Releasing cgimage though is a problem in my code, the iteration is completed but right afterwards an exception is thrown. I will keep you up to date. Thank for looking through my code and for your suggestion. – BCI May 14 '14 at 08:20
The struct cgImage of type cgImageRef should not be released, when it is not owned by the caller. The code I have written cgImage is never created. So there probably is another difference between our codes, you most likely have created this struct at some point. Could you tell me where and how? Thanks – BCI May 15 '14 at 11:06
I need to mention that releasing cgImage has a positive effect, despite the crashing. It crashes once the iteration is done and it does not matter how many times it iterated, respective how many images are used for movie generation. So you did put me on the right track in some way. – BCI May 15 '14 at 11:17
thanks for your code sample, it helped me, but it was not the easy solution for me. – BCI May 19 '14 at 09:16
Hi again Tim ... would you mind sharing the full code for converting images to a video? If not, thanks anyway for sharing this fragment! – Crashalot Mar 23 '16 at 00:45
1

@TimBull finished porting a Swift version if you want to check it out, incorporates some of the AVAssetWriter code you generously supplied before: http://stackoverflow.com/questions/3741323/how-do-i-export-uiimage-array-as-a-movie/36297656#36297656. Thanks again for all your help! – Crashalot Mar 30 '16 at 00:44

BCI · Accepted Answer · 2014-05-19T09:14:10.483

Finally I found the solution to my problem, there were 2 points I had to change in my code.

I changed the parameter type of the method (CVPixelBufferRef) pixelBufferFromImage: (UIImage*) image from CGImageRef to UIImage. Now the reason for this is mainly to simplify the code so the coming correction is easier to implement.
Autoreleasepool is introduced to this method. Now this is the actual key to the solution. CGImageRef cgimage = [image CGImage]; and all other components of the method must be included in the Autoreleasepool.

The code looks like this.

- (CVPixelBufferRef) pixelBufferFromImage: (UIImage*) image withOrientation:(ImageOrientation)orientation
{
 @autoreleasepool
 {    
 CGImageRef cgimage = [image CGImage];

 NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                         [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                         [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                         nil];

 CVPixelBufferRef pxbuffer = NULL;

 float width =  CGImageGetWidth(cgimage);
 float height = CGImageGetHeight(cgimage);


 CVPixelBufferCreate(kCFAllocatorDefault,width,
                                      height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef)(options),
                                      &pxbuffer);

 CVPixelBufferLockBaseAddress(pxbuffer, 0);

 void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);

 NSParameterAssert(pxdata != NULL);

 CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
 CGContextRef context = CGBitmapContextCreate(pxdata, width,
                                             height, 8, 4*width, rgbColorSpace,
                                              (CGBitmapInfo)kCGImageAlphaNoneSkipFirst);


  CGContextConcatCTM(context, CGAffineTransformMakeRotation(-M_PI/2));

  CGContextDrawImage(context, CGRectMake(-height, 0, height, width), cgimage);

  CGColorSpaceRelease(rgbColorSpace);
  CGContextRelease(context);

  CVPixelBufferUnlockBaseAddress(pxbuffer, 0);

  return pxbuffer;
 }
}

With this solution a hd movie of more than 2000 images is generated at a rather slow speed but it seems to be very reliable, which is most important.

Generate movie with UIImages using AVFoundation

2 Answers2