Drawing a CIImage is too slow

Question

I'm creating an app that requires real-time application of filters to images. Converting the UIImage to a CIImage, and applying the filters are both extremely fast operations, yet it takes too long to convert the created CIImage back to a CGImageRef and display the image (1/5 of a second, which is actually a lot if editing needs to be real-time).

The image is about 2500 by 2500 pixels big, which is most likely part of the problem

Currently, I'm using

let image: CIImage //CIImage with applied filters
let eagl = EAGLContext(API: EAGLRenderingAPI.OpenGLES2)
let context = CIContext(EAGLContext: eagl, options: [kCIContextWorkingColorSpace : NSNull()])

//this line takes too long for real-time processing
let cg: CGImage = context.createCGImage(image, fromRect: image.extent)

I've looked into using EAGLContext.drawImage()

context.drawImage(image, inRect: destinationRect, fromRect: image.extent)

Yet I can't find any solid documentation on exactly how this is done, or if it would be any faster

Is there any faster way to display a CIImage to the screen (either in a UIImageView, or directly on a CALayer)? I would like to avoid decreasing the image quality too much, because this may be noticeable to the user.

score 16 · Answer 1 · answered May 27 '16 at 04:16

16

It may be worth considering Metal and displaying with a MTKView.

You'll need a Metal device which can be created with MTLCreateSystemDefaultDevice(). That's used to create a command queue and Core Image context. Both these objects are persistent and quite expensive to instantiate, so ideally should be created once:

lazy var commandQueue: MTLCommandQueue =
{
    return self.device!.newCommandQueue()
}()

lazy var ciContext: CIContext =
{
    return CIContext(MTLDevice: self.device!)
}()

You'll also need a color space:

let colorSpace = CGColorSpaceCreateDeviceRGB()!

When it comes to rendering a CIImage, you'll need to create a short lived command buffer:

let commandBuffer = commandQueue.commandBuffer()

You'll want to render your CIImage (let's call it image) to the currentDrawable?.texture of a MTKView. If that's bound to targetTexture, the rendering syntax is:

    ciContext.render(image,
        toMTLTexture: targetTexture,
        commandBuffer: commandBuffer,
        bounds: image.extent,
        colorSpace: colorSpace)

    commandBuffer.presentDrawable(currentDrawable!)

    commandBuffer.commit()

I have a working version here.

Hope that helps!

Simon

answered May 27 '16 at 04:16

Flex Monkey

3,583
17
19

I found this the most performant solution. Note that in my code, since I'm not subclassing MTKView, I had to call `draw()` on the `MTKView` immediately after calling `commit()` and also had to set the view's `device` to the one created and its `framebufferOnly` to `false`. – Ash Jan 12 '18 at 09:02
It's also worth looking at `CIRenderDestination` and the `startTask` method on `CIContext`. – Flex Monkey Jan 12 '18 at 09:16
I have noticed one annoyance with this function that I'm currently working on addressing, and that's if the view resizes, the texture size of the backing drawable always seems to be one step behind the new size and I cannot seem to find a way to force it to update properly. As it stands, your own view negates this issue using stretching, but I'm creating a graphical application that requires high quality output and I want the image to be rendered at exactly the size of the containing view. – Ash Jan 12 '18 at 10:52
1

OK, I have a solution - you might want to put this into your own linked version. Simply rename the `renderImage()` function to `override func draw()`, calling `super.draw()` at the end of it, and in the `didSet` of `image` set `needsDisplay` to `true` instead of calling the old function. – Ash Jan 12 '18 at 11:01
It's extremely important to call draw() after commandBuffer.commit() or else you'll get a bunch of errors in your console and your app will freeze for several seconds before recovering! – Chewie The Chorkie Sep 10 '18 at 21:02
How do you render transparency of a png? i.e. watermark – masaldana2 Dec 04 '18 at 05:24
you're the best Simon – Tom Roggero May 29 '19 at 18:05

Jojodmo · Accepted Answer · 2017-01-29T02:05:14.797

I ended up using the context.drawImage(image, inRect: destinationRect, fromRect: image.extent) method. Here's the image view class that I created

import Foundation
//GLKit must be linked and imported
import GLKit

class CIImageView: GLKView{
    var image: CIImage?
    var ciContext: CIContext?

    //initialize with the frame, and CIImage to be displayed
    //(or nil, if the image will be set using .setRenderImage)
    init(frame: CGRect, image: CIImage?){
        super.init(frame: frame, context: EAGLContext(API: EAGLRenderingAPI.OpenGLES2))

        self.image = image
        //Set the current context to the EAGLContext created in the super.init call
        EAGLContext.setCurrentContext(self.context)
        //create a CIContext from the EAGLContext
        self.ciContext = CIContext(EAGLContext: self.context)
    }

    //for usage in Storyboards
    required init?(coder aDecoder: NSCoder){
        super.init(coder: aDecoder)

        self.context = EAGLContext(API: EAGLRenderingAPI.OpenGLES2)
        EAGLContext.setCurrentContext(self.context)
        self.ciContext = CIContext(EAGLContext: self.context)
    }

    //set the current image to image
    func setRenderImage(image: CIImage){
        self.image = image

        //tell the processor that the view needs to be redrawn using drawRect()
        self.setNeedsDisplay()
    }

    //called automatically when the view is drawn
    override func drawRect(rect: CGRect){
        //unwrap the current CIImage
        if let image = self.image{
            //multiply the frame by the screen's scale (ratio of points : pixels),
            //because the following .drawImage() call uses pixels, not points
            let scale = UIScreen.mainScreen().scale
            let newFrame = CGRectMake(rect.minX, rect.minY, rect.width * scale, rect.height * scale)

            //draw the image
            self.ciContext?.drawImage(
                image,
                inRect: newFrame,
                fromRect: image.extent
             )
        }
    }   
}

Then, to use it, simply

let myFrame: CGRect //frame in self.view where the image should be displayed
let myImage: CIImage //CIImage with applied filters

let imageView: CIImageView = CIImageView(frame: myFrame, image: myImage)
self.view.addSubview(imageView)

Resizing the UIImage to the screen size before converting it to a CIImage also helps. It speeds things up a lot in the case of high quality images. Just make sure to use the full-size image when actually saving it.

Thats it! Then, to update the image in the view

imageView.setRenderImage(newCIImage)
//note that imageView.image = newCIImage won't work because
//the view won't be redrawn

you could use didSet on image property: `var image: CIImage? { didSet { self.setNeedsDisplay() } }` Then whole setRenderImage method becomes obsolete. — Marcin Zbijowski, Sep 28 '16 at 12:37
You must set the context in init?(coder:) (like you did it in init(frame)) in order for this to work from storyboard. — Jovan Stankovic, Jan 28 '17 at 23:33
Is this recommended now that GLKView has been deprecated since iOS 12? — Michael Forrest, Oct 08 '20 at 10:46
@MichaelForrest not recommended of course. You need to use Metal instead. — Kegham K., Sep 20 '22 at 00:14

score 2 · Answer 3 · answered May 26 '16 at 10:41

You can use GlkView and render as you said with context.drawImage() :

let glView = GLKView(frame: superview.bounds, context: EAGLContext(API: .OpenGLES2))
let context = CIContext(EAGLContext: glView.context)

After your processing render the image :

glView.bindDrawable()
context.drawImage(image, inRect: destinationRect, fromRect: image.extent)
glView.display()

score 0 · Answer 4 · answered May 26 '16 at 03:13

That is a pretty big image so that's definitely part of it. I'd recommend looking at GPUImage for doing single image filters. You can skip over using CoreImage altogether.

let inputImage:UIImage  = //... some image
let stillImageSource = GPUImagePicture(image: inputImage)
let filter = GPUImageSepiaFilter()
stillImageSource.addTarget(filter)
filter.useNextFrameForImageCapture()
stillImageSource.processImage()

Drawing a CIImage is too slow

4 Answers4